Distributed data transformation system

ABSTRACT

A computing system transforms variable values in a dataset using a transformation flow definition applied in parallel. The transformation flow definition indicates flow variables and transformation phases to apply to the flow variables. A computation is defined for each variable and for each transformation phase. A phase internal parameter value is computed for each defined computation from observation vectors read from the dataset. A current variable, a first variable value, a first transformation phase, the phase internal parameter value, and a current transformation phase are selected based on an observation vector read from the dataset. A result value is computed by executing the transformation function with the phase internal parameter value and the first variable value. The computed result value is output to a transformed input dataset. The process is repeated for each variable, transformation phase, and observation vector.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is also a continuation of U.S. patentapplication Ser. No. 15/876,543 that was filed Jan. 22, 2018, and issuedas U.S. Pat. No. 10,025,813 on Jul. 17, 2018, the entire contents ofwhich are hereby incorporated by reference. U.S. patent application Ser.No. 15/876,543 claims the benefit of 35 U.S.C. § 119(e) to U.S.Provisional Patent Application No. 62/485,247 filed Apr. 13, 2017, theentire contents of which are hereby incorporated by reference.

BACKGROUND

One of the common characteristics of many modern datasets is highdimensionality along with low signal-to-noise ratio due to a potentiallylarge number of irrelevant variables. Quantifying data-quality issuesusing statistical data quality metrics such as missing rate,cardinality, etc. is the first task in predictive modelling of adataset. As a result, variable (feature) transformation aimed atincreasing model performance is a significant part of a predictivemodelling workflow. However, high dimensionality precludes aninteractive variable-by-variable analysis and transformation. To handlethis issue of scale (high dimensionality), practitioners consider dataquality issues iteratively. For example, variables with a high-rate ofmissing values can be identified and addressed. Variables with ahigh-skew can then be identified and addressed. However, this approachprecludes the effective utilization of prescriptions that can treatmultiple data quality problems at the same time. In addition, thisapproach is prone to significant bias, especially in cases whereimputation is applied to variables with high missing rate. Automateddata preprocessing with meta-learning machine learning systems isanother potential solution to the scale issue. However, currentmeta-learning systems use dataset features that are based solely onindividual data quality metrics, and do not take interactions betweendata quality metrics into consideration. This approach finds itchallenging to retain sufficient information that describes the dataset,which is a critical step for meta-learning based approaches.

SUMMARY

In an example embodiment, a computer-readable medium is provided havingstored thereon computer-readable instructions that when executed by acomputing device, cause the computing device to provide analysis ofdistributed data and grouping of variables in support of analytics. Areceived transformation flow definition indicates one or more flowvariables and one or more transformation phases to apply to the one ormore flow variables. A statistical computation is defined for eachvariable of the one or more variables and for each transformation phaseof the one or more transformation phases. A phase internal parametervalue is computed for each defined statistical computation from aplurality of observation vectors read from an input dataset. Eachobservation vector of the plurality of observation vectors includes aplurality of values. Each value of the plurality of values is associatedwith a different variable to define a plurality of variables. Eachvariable of the one or more variables is associated with one of thedefined plurality of variables. (a) an observation vector is read fromthe input dataset to define a first variable value for each variable ofthe one or more flow variables. (b) a variable of the one or more flowvariables is selected as a current variable. (c) a current value equalto the defined first variable value is defined for the current variable.(d) a first transformation phase of the one or more transformationphases is selected. (e) the computed phase internal parameter value forthe current variable and the current transformation phase is selected.(f) a transformation function is defined with the selected phaseinternal parameter value based on the current transformation phase. (g)a result value is computed by executing the defined transformationfunction with the defined current value. (h) (e) to (g) are repeatedwith each remaining transformation phase of the one or moretransformation phases as the current transformation phase and with thedefined current value equal to the computed result value. (i) thecomputed result value is output to a transformed input dataset. (j) (c)to (i) are repeated with each remaining variable of the one or more flowvariables as the current variable. (a) to (j) are repeated with eachremaining observation vector from the input dataset.

In another example embodiment, a system is provided. The systemincludes, but is not limited to, a processor and a computer-readablemedium operably coupled to the processor. The computer-readable mediumhas instructions stored thereon that, when executed by the processor,cause the system to provide analysis of distributed data and grouping ofvariables in support of analytics.

In yet another example embodiment, a method of providing analysis ofdistributed data and grouping of variables in support of analytics isprovided.

Other principal features of the disclosed subject matter will becomeapparent to those skilled in the art upon review of the followingdrawings, the detailed description, and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the disclosed subject matter will hereafterbe described referring to the accompanying drawings, wherein likenumerals denote like elements.

FIG. 1 depicts a block diagram of a data analysis and transformationsystem in accordance with an illustrative embodiment.

FIG. 2 depicts a block diagram of a user device of the data analysis andtransformation system of FIG. 1 in accordance with an illustrativeembodiment.

FIG. 3 depicts a block diagram of a controller device of the dataanalysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 4 depicts a block diagram of a worker device of the data analysisand transformation system of FIG. 1 in accordance with an illustrativeembodiment.

FIG. 5 depicts a flow diagram illustrating examples of operationsperformed by the user device of FIG. 2 in support of data analysis andgrouping in accordance with an illustrative embodiment.

FIG. 6 depicts a flow diagram illustrating examples of operationsperformed by the controller device of FIG. 3 in support of data analysisand grouping in accordance with an illustrative embodiment.

FIGS. 7A and 7B depict a flow diagram illustrating examples ofoperations performed by the worker device of FIG. 4 in support of dataanalysis and grouping in accordance with an illustrative embodiment.

FIGS. 8A to 8C depict user interface options provided by the user deviceof FIG. 2 in accordance with an illustrative embodiment.

FIG. 9 depicts data analysis results presented by the user device ofFIG. 2 in accordance with an illustrative embodiment.

FIG. 10 illustrates a variable grouping result tree determined by thedata analysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 11 depicts a second block diagram of the user device of the dataanalysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 12 depicts a second block diagram of a controller device of thedata analysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 13 depicts a second block diagram of a worker device of the dataanalysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 14 depicts a flow diagram illustrating examples of operationsperformed by the user device of FIG. 2 in support of data transformationin accordance with an illustrative embodiment.

FIGS. 15A and 15B depicts a flow diagram illustrating examples ofoperations performed by the controller device of FIG. 3 in support ofdata transformation in accordance with an illustrative embodiment.

FIGS. 16A, 16B, and 16C depicts a flow diagram illustrating examples ofoperations performed by the worker device of FIG. 4 in support of datatransformation in accordance with an illustrative embodiment.

FIG. 17 depicts a third block diagram of the user device of the dataanalysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 18 depicts a third block diagram of a controller device of the dataanalysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 19 depicts a third block diagram of a worker device of the dataanalysis and transformation system of FIG. 1 in accordance with anillustrative embodiment.

FIG. 20 depicts a flow diagram illustrating examples of operationsperformed by the user device of FIG. 2 in support of high-cardinality(high-C) data transformation in accordance with an illustrativeembodiment.

FIGS. 21A and 21B depicts a flow diagram illustrating examples ofoperations performed by the controller device of FIG. 3 in support ofhigh-C data transformation in accordance with an illustrativeembodiment.

FIGS. 22A and 22B depicts a flow diagram illustrating examples ofoperations performed by the worker device of FIG. 4 in support of high-Cdata transformation in accordance with an illustrative embodiment.

FIG. 23 depicts a block diagram of a model training device in accordancewith an illustrative embodiment.

FIG. 24 depicts a flow diagram illustrating examples of operationsperformed by the model training device of FIG. 23 in accordance with anillustrative embodiment.

FIG. 25 depicts a block diagram of a prediction device in accordancewith an illustrative embodiment.

FIG. 26 depicts a flow diagram illustrating examples of operationsperformed by the prediction device of FIG. 25 in accordance with anillustrative embodiment.

DETAILED DESCRIPTION

Referring to FIG. 1, a block diagram of a data analysis andtransformation system 100 is shown in accordance with an illustrativeembodiment. In an illustrative embodiment, data analysis andtransformation system 100 may include a user system 102, a controllerdevice 104, a worker system 106, and a network 108. Each of user system102, controller device 104, and worker system 106 may be composed of oneor more discrete computing devices in communication through network 108.User system 102 and controller device 104 may be integrated into asingle computing device.

Data analysis and transformation system 100 performs automatic variableanalysis and grouping in two data passes of an input dataset. Dataanalysis and transformation system 100 provides effective visualizationof data quality problems in modern datasets that are typicallycharacterized by large dimensions. Data analysis and transformationsystem 100 further consumes the data analytics to perform a plurality ofvariable transformation flows simultaneously in a minimum of data passeswith a minimum of calculations so that the user can explore multipleoptions for transforming one or more variables of the input dataset. Theresulting transformed dataset that may include multiple datatransformations of the same data can be input to a model training systemto train one or more models that describe various characteristics of thetransformed dataset. The trained model can be applied to new data topredict a characteristic of or to monitor the new data foridentification of occurrence of an event.

Network 108 may include one or more networks of the same or differenttypes. Network 108 can be any type of wired and/or wireless public orprivate network including a cellular network, a local area network, awide area network such as the Internet or the World Wide Web, etc.Network 108 further may comprise sub-networks and consist of any numberof communication devices.

The one or more computing devices of user system 102 may includecomputing devices of any form factor such as a desktop 110, a smartphone 112, a server computer 114, a laptop 116, a personal digitalassistant, an integrated messaging device, a tablet computer, etc. Usersystem 102 can include any number and any combination of form factors ofcomputing devices that may be organized into subnets. In general, a“server” computer may include faster processors, additional processors,more disk memory, and/or more random access memory (RAM) than anothercomputer form factor and support multi-threading as understood by aperson of skill in the art. The computing devices of user system 102 maysend and receive signals through network 108 to/from another of the oneor more computing devices of user system 102 and/or to/from controllerdevice 104. The one or more computing devices of user system 102 maycommunicate using various transmission media that may be wired and/orwireless as understood by those skilled in the art. The one or morecomputing devices of user system 102 may be geographically dispersedfrom each other and/or co-located.

For illustration, referring to FIG. 2, a block diagram of a user device200 is shown in accordance with an example embodiment. User device 200is an example computing device of user system 102. For example, each ofdesktop 110, smart phone 112, server computer 114, and laptop 116 may bean instance of user device 200. User device 200 may include an inputinterface 202, an output interface 204, a communication interface 206, anon-transitory computer-readable medium 208, a processor 210, a dataanalysis application 222, and data analysis results 223. Each computingdevice of user system 102 may be executing data analysis application 222of the same or different type.

Referring again to FIG. 1, controller device 104 can include any formfactor of computing device. For illustration, FIG. 1 representscontroller device 104 as a server computer. Controller device 104 maysend and receive signals through network 108 to/from user system 102and/or to/from worker system 106. Controller device 104 may communicateusing various transmission media that may be wired and/or wireless asunderstood by those skilled in the art. Controller device 104 may beimplemented on a plurality of computing devices of the same or differenttype. Data analysis and transformation system 100 further may include aplurality of controller devices that communicate with user system 102and worker system 106.

For illustration, referring to FIG. 3, a block diagram of controllerdevice 104 is shown in accordance with an illustrative embodiment.Controller device 104 may include a second input interface 302, a secondoutput interface 304, a second communication interface 306, a secondnon-transitory computer-readable medium 308, a second processor 310, acontroller data analysis application 312, policy parameter values 314,and data analysis results 316. Controller device 104 may executecontroller data analysis application 312 that creates data analysisresults 316 based on the input dataset that may be distributed acrossthe computing devices of worker system 106 and on policy parametervalues 314 that may be defined by a user of user device 200.

Referring again to FIG. 1, the one or more computing devices of workersystem 106 may include computers of any form factor such as a desktop, aserver, a laptop, etc. For example, in the illustrative embodiment,worker system 106 includes a first server computer 118-a, . . . , and annth server computer 118-n. Each server computer may support use of aplurality of threads. The computing devices of worker system 106 maysend and receive signals through network 108 to/from controller device104 and/or to/from another computing device of worker system 106. Theone or more computing devices of worker system 106 may be geographicallydispersed from each other and/or co-located. The one or more computingdevices of worker system 106 may communicate using various transmissionmedia that may be wired and/or wireless as understood by those skilledin the art.

For illustration, referring to FIG. 4, a block diagram of a workerdevice 400 is shown in accordance with an example embodiment. Workerdevice 400 is an example computing device of worker system 106. Forexample, each of first server computer 118-a, . . . , and nth servercomputer 118-n may be an instance of worker device 400. Worker device400 may include a third input interface 402, a third output interface404, a third communication interface 406, a third non-transitorycomputer-readable medium 408, a third processor 410, a worker dataanalysis application 412, an input data subset 414, a subset statisticsdataset 416, and variable grouping data 418.

Referring again to FIG. 2, each user device 200 of user system 102 mayinclude the same or different components and combinations of components.Fewer, different, and additional components may be incorporated intouser device 200.

Input interface 202 provides an interface for receiving information forentry into user device 200 as understood by those skilled in the art.Input interface 202 may interface with various input technologiesincluding, but not limited to, a keyboard 212, a mouse 214, a display216, a track ball, a keypad, one or more buttons, etc. to allow the userto enter information into user device 200 or to make selectionspresented in a user interface displayed on display 216. The sameinterface may support both input interface 202 and output interface 204.For example, display 216 includes a touch screen that accepts input fromthe user and that presents output to the user. User device 200 may haveone or more input interfaces that use the same or a different inputinterface technology. The input interface technology further may beaccessible by user device 200 through communication interface 206.

Output interface 204 provides an interface for outputting informationfor review by a user of user device 200. For example, output interface204 may interface with various output technologies including, but notlimited to, display 216, a speaker 218, a printer 220, etc. User device200 may have one or more output interfaces that use the same or adifferent interface technology. The output interface technology furthermay be accessible by user device 200 through communication interface206.

Communication interface 206 provides an interface for receiving andtransmitting data between devices using various protocols, transmissiontechnologies, and media as understood by those skilled in the art.Communication interface 206 may support communication using varioustransmission media that may be wired and/or wireless. User device 200may have one or more communication interfaces that use the same or adifferent communication interface technology. For example, user device200 may support communication using an Ethernet port, a Bluetoothantenna, a telephone jack, a USB port, etc. Data and messages may betransferred between user device 200 and controller device 104 usingcommunication interface 206.

Computer-readable medium 208 is a non-transitory electronic holdingplace or storage for information so the information can be accessed byprocessor 210 as understood by those skilled in the art.Computer-readable medium 208 can include, but is not limited to, anytype of random access memory (RAM), any type of read only memory (ROM),any type of flash memory, etc. such as magnetic storage devices (e.g.,hard disk, floppy disk, magnetic strips, . . . ), optical disks (e.g.,compact disc (CD), digital versatile disc (DVD), . . . ), smart cards,flash memory devices, etc. User device 200 may have one or morecomputer-readable media that use the same or a different memory mediatechnology. For example, computer-readable medium 208 may includedifferent types of computer-readable media that may be organizedhierarchically to provide efficient access to the data stored therein asunderstood by a person of skill in the art. As an example, a cache maybe implemented in a smaller, faster memory that stores copies of datafrom the most frequently/recently accessed main memory locations toreduce an access latency. User device 200 also may have one or moredrives that support the loading of a memory media such as a CD or DVD,an external hard drive, etc. One or more external hard drives furthermay be connected to user device 200 using communication interface 106and/or output interface 204.

Processor 210 executes instructions as understood by those skilled inthe art. The instructions may be carried out by a special purposecomputer, logic circuits, or hardware circuits. Processor 210 may beimplemented in hardware and/or firmware. Processor 210 executes aninstruction, meaning it performs/controls the operations called for bythat instruction. The term “execution” is the process of running anapplication or the carrying out of the operation called for by aninstruction. The instructions may be written using one or moreprogramming language, scripting language, assembly language, etc.Processor 210 operably couples with input interface 202, with outputinterface 204, with communication interface 206, and withcomputer-readable medium 208 to receive, to send, and to processinformation. Processor 210 may retrieve a set of instructions from apermanent memory device and copy the instructions in an executable formto a temporary memory device that is generally some form of RAM. Userdevice 200 may include a plurality of processors that use the same or adifferent processing technology.

Data analysis application 222 performs operations associated withrequesting analysis of the input dataset so that the user can understandthe data stored in the input dataset. The operations may be implementedusing hardware, firmware, software, or any combination of these methods.Referring to the example embodiment of FIG. 2, data analysis application222 is implemented in software (comprised of computer-readable and/orcomputer-executable instructions) stored in computer-readable medium 208and accessible by processor 210 for execution of the instructions thatembody the operations of data analysis application 222. Data analysisapplication 222 may be written using one or more programming languages,assembly languages, scripting languages, etc.

Data analysis application 222 may be implemented as a Web application.For example, data analysis application 222 may be configured to receivehypertext transport protocol (HTTP) responses and to send HTTP requests.The HTTP responses may include web pages such as hypertext markuplanguage (HTML) documents and linked objects generated in response tothe HTTP requests. Each web page may be identified by a uniform resourcelocator (URL) that includes the location or address of the computingdevice that contains the resource to be accessed in addition to thelocation of the resource on that computing device. The type of file orresource depends on the Internet application protocol such as the filetransfer protocol, HTTP, H.323, etc. The file accessed may be a simpletext file, an image file, an audio file, a video file, an executable, acommon gateway interface application, a Java applet, an XML file, or anyother type of file supported by HTTP.

Data analysis application 222 may be integrated with other analytictools. As an example, data analysis application 222 may be part of anintegrated data analytics software application and/or softwarearchitecture such as that offered by SAS Institute Inc. of Cary, N.C.,USA. For example, data analysis application 222 may be part of SAS®Enterprise Miner™ developed and provided by SAS Institute Inc. of Cary,N.C., USA that may be used to create highly accurate predictive anddescriptive models based on analysis of vast amounts of data from acrossan enterprise. Merely for further illustration, data analysisapplication 222 may be implemented using or integrated with one or moreSAS software tools such as Base SAS, SAS/STAT®, SAS® High PerformanceAnalytics Server, SAS® LASR™, SAS® In-Database Products, SAS® ScalablePerformance Data Engine, SAS/OR®, SAS/ETS®, SAS® Inventory Optimization,SAS® Inventory Optimization Workbench, SAS® Visual Data Mining andMachine Learning, SAS® Visual Analytics, SAS® Viya™, SAS In-MemoryStatistics for Hadoop®, SAS® Forecast Server, all of which are developedand provided by SAS Institute Inc. of Cary, N.C., USA. Data mining isapplicable in a wide variety of industries.

Referring to FIG. 3, fewer, different, or additional components may beincorporated into controller device 104. Second input interface 302provides the same or similar functionality as that described withreference to input interface 202 of user device 200 though referring tocontroller device 104. Second output interface 304 provides the same orsimilar functionality as that described with reference to outputinterface 204 of user device 200 though referring to controller device104. Second communication interface 306 provides the same or similarfunctionality as that described with reference to communicationinterface 206 of user device 200 though referring to controller device104. Data and messages may be transferred between controller device 104and user device 200 and/or worker device 400 using second communicationinterface 306. Second computer-readable medium 308 provides the same orsimilar functionality as that described with reference tocomputer-readable medium 208 of user device 200 though referring tocontroller device 104. Second processor 310 provides the same or similarfunctionality as that described with reference to processor 210 of userdevice 200 though referring to controller device 104.

Controller data analysis application 312 performs operations associatedwith performing variable statistical analysis and grouping of the inputdataset based on inputs provided from user device 200 using thecomputing devices of worker system 106. The input dataset may bedistributed across the computing devices of worker system 106. Theoperations may be implemented using hardware, firmware, software, or anycombination of these methods. Referring to the example embodiment ofFIG. 3, controller data analysis application 312 is implemented insoftware (comprised of computer-readable and/or computer-executableinstructions) stored in second computer-readable medium 308 andaccessible by second processor 310 for execution of the instructionsthat embody the operations of controller data analysis application 312.Controller data analysis application 312 may be written using one ormore programming languages, assembly languages, scripting languages,etc. Controller data analysis application 312 may be implemented as aWeb application.

Controller data analysis application 312 may be integrated with otheranalytic tools. As an example, controller data analysis application 312may be part of an integrated data analytics software application and/orsoftware architecture such as that offered by SAS Institute Inc. ofCary, N.C., USA. For example, controller data analysis application 312may be part of SAS® Enterprise Miner™ developed and provided by SASInstitute Inc. of Cary, N.C., USA. Merely for further illustration,controller data analysis application 312 may be implemented using orintegrated with one or more SAS software tools such as Base SAS,SAS/STAT®, SAS® High Performance Analytics Server, SAS® LASR™, SAS®In-Database Products, SAS® Scalable Performance Data Engine, SAS/OR®,SAS/ETS®, SAS® Inventory Optimization, SAS® Inventory OptimizationWorkbench, SAS® Visual Data Mining and Machine Learning, SAS® VisualAnalytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®, SAS®Forecast Server, all of which are developed and provided by SASInstitute Inc. of Cary, N.C., USA.

Referring to FIG. 4, fewer, different, and additional components may beincorporated into worker device 400. Each worker device 400 of workersystem 106 may include the same or different components or combinationof components.

Third input interface 402 provides the same or similar functionality asthat described with reference to input interface 202 of user device 200though referring to worker device 400. Third output interface 404provides the same or similar functionality as that described withreference to output interface 204 of user device 200 though referring toworker device 400. Third communication interface 406 provides the sameor similar functionality as that described with reference tocommunication interface 206 of user device 200 though referring toworker device 400. Data and messages may be transferred between workerdevice 400 and another computing device of worker system 106 and/orcontroller device 104 using third communication interface 406. Thirdcomputer-readable medium 408 provides the same or similar functionalityas that described with reference to computer-readable medium 208 of userdevice 200 though referring to worker device 400. Third processor 410provides the same or similar functionality as that described withreference to processor 210 of user device 200 though referring to workerdevice 400.

Worker data analysis application 412 performs variable statisticalanalysis and grouping of input data subset 414 based on inputs fromcontroller device 104 to define subset statistics dataset 416 andvariable grouping data 418 that is returned, or otherwise madeavailable, to controller device 104. Worker data analysis application412 may be integrated with other analytic tools. As an example, workerdata analysis application 412 may be part of an integrated dataanalytics software application and/or software architecture such as thatoffered by SAS Institute Inc. of Cary, N.C., USA. For example, workerdata analysis application 412 may be part of SAS® Enterprise Miner™developed and provided by SAS Institute Inc. of Cary, N.C., USA. Merelyfor further illustration, worker data analysis application 412 may beimplemented using or integrated with one or more SAS software tools suchas Base SAS, SAS/STAT®, SAS® High Performance Analytics Server, SAS®LASR™, SAS® In-Database Products, SAS® Scalable Performance Data Engine,SAS/OR®, SAS/ETS®, SAS® Inventory Optimization, SAS® InventoryOptimization Workbench, SAS® Visual Data Mining and Machine Learning,SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®,SAS® Forecast Server, all of which are developed and provided by SASInstitute Inc. of Cary, N.C., USA.

Data analysis application 222, controller data analysis application 312,and worker data analysis application 412 may be the same or differentapplications that are integrated in various manners to perform variablestatistical analysis and grouping of the input dataset distributedacross worker system 106. A subset of the input dataset may further maybe stored on controller device 104.

The input dataset may include, for example, a plurality of rows and aplurality of columns. The plurality of rows may be referred to asobservation vectors or records (observations), and the columns may bereferred to as variables (features). The input dataset may betransposed. The input dataset may include supervised (target variable)and/or unsupervised data. The plurality of variables may define multipledimensions or features for each observation vector. An observationvector x_(i) may include a value for each of the plurality of variablesassociated with the observation i. One or more values may be missingfrom one or more observation vectors and is referred to herein asmissing data or missing data values. Each variable of the plurality ofvariables may describe a characteristic of a physical object. Forexample, if the input dataset includes data related to operation of avehicle, the variables may include an oil pressure, a speed, a gearindicator, a gas tank level, a tire pressure for each tire, an enginetemperature, a radiator level, etc. The input dataset may include datacaptured as a function of time for one or more physical objects. Asanother example, the input dataset may include data related to images,where each row includes the pixels that define a single image. Theimages may be of any item for which image recognition or classificationmay be performed including, but not limited to, faces, objects,alphanumeric letters, terrain, plants, animals, etc.

The data stored in the input dataset may be generated by and/or capturedfrom a variety of sources including one or more sensors of the same ordifferent type, one or more computing devices, etc. The data stored inthe input dataset may be received directly or indirectly from the sourceand may or may not be preprocessed in some manner. For example, the datamay be preprocessed using an event stream processor such as the SAS®Event Stream Processing Engine (ESPE), developed and provided by SASInstitute Inc. of Cary, N.C., USA. As used herein, the data may includeany type of content represented in any computer-readable format such asbinary, alphanumeric, numeric, string, markup language, etc. The datamay be organized using delimited fields, such as comma or spaceseparated fields, fixed width fields, using a SAS® dataset, etc. The SASdataset may be a SAS® file stored in a SAS® library that a SAS® softwaretool creates and processes. The SAS dataset contains data values thatare organized as a table of observations (rows) and variables (columns)that can be processed by one or more SAS software tools.

The input dataset may be stored on computer-readable medium 208, onsecond computer-readable medium 308, and/or on third computer-readablemedium 408 of each worker device 400. In an illustrative embodiment, theinput dataset may be distributed and loaded on each thirdcomputer-readable medium 408 of worker system 106. Data stored in theinput dataset may be sensor measurements or signal values captured by asensor such as a camera, may be generated or captured in response tooccurrence of an event or a transaction, generated by a device such asin response to an interaction by a user with the device, etc. The datastored in the input dataset may include any type of content representedin any computer-readable format such as binary, alphanumeric, numeric,string, markup language, etc. The content may include textualinformation, graphical information, image information, audioinformation, numeric information, etc. that further may be encoded usingvarious encoding techniques as understood by a person of skill in theart. The data stored in the input dataset may be captured at differenttime points periodically, intermittently, when an event occurs, etc. Oneor more columns of the input dataset may include a time and/or datevalue.

The input dataset may include data captured under normal operatingconditions of a physical object. The input dataset may include datacaptured at a high data rate such as 200 or more observations per secondfor one or more physical objects. For example, data stored in the inputdataset may be generated as part of the Internet of Things (IoT), wherethings (e.g., machines, devices, phones, sensors, smart meters forenergy, personal wearable devices, health monitoring devices, autonomousvehicle devices, robotic components, identification devices, etc.) canbe connected to networks and the data from these things collected andprocessed within the things and/or external to the things before beingstored in the input dataset. For example, the IoT can include sensors inmany different devices and types of devices, and high value analyticscan be applied to identify hidden relationships and to drive increasedefficiencies. This can apply to both big data analytics and real-timeanalytics. Some of these devices may be referred to as edge devices, andmay involve edge computing circuitry. These devices may provide avariety of stored or generated data, such as network data or dataspecific to the network devices themselves. Again, some data may beprocessed with an ESPE, which may reside in the cloud or in an edgedevice before being stored in the input dataset.

The input dataset may be stored using various structures as known tothose skilled in the art including one or more files of a file system, arelational database, one or more tables of a system of tables, astructured query language database, etc. Controller device 104 maycoordinate access to the input dataset that is distributed across workersystem 106. For example, the input dataset may be stored in a cubedistributed across worker system 106 that forms a grid of computers asunderstood by a person of skill in the art. As another example, theinput dataset may be stored in a multi-node Hadoop® cluster. Forinstance, Apache™ Hadoop® is an open-source software framework fordistributed computing supported by the Apache Software Foundation. Asanother example, the input dataset may be stored in worker system 106that forms a cloud of computers and is accessed using cloud computingtechnologies, as understood by a person of skill in the art. The SAS®LASR™ Analytic Server may be used as an analytic platform to enablemultiple users to concurrently access data stored in the input dataset.The SAS® Viya™ open, cloud-ready, in-memory architecture also may beused as an analytic platform to enable multiple users to concurrentlyaccess data stored in the input dataset. SAS Cloud Analytic Services(CAS) may be used as an analytic server with associated cloud servicesin SAS Viya. Some systems may use SAS In-Memory Statistics for Hadoop®to read big data once and analyze it several times by persisting itin-memory for the entire session. Some systems may be of other types andconfigurations.

Referring to FIG. 5, example operations associated with data analysisapplication 222 are described. Additional, fewer, or differentoperations may be performed depending on the embodiment. The order ofpresentation of the operations of FIG. 5 is not intended to be limiting.A user can interact with one or more user interface windows presented tothe user in a display under control of data analysis application 222independently or through a browser application in an order selectable bythe user. Although some of the operational flows are presented insequence, the various operations may be performed in variousrepetitions, concurrently, and/or in other orders than those that areillustrated. For example, a user may execute data analysis application222, which causes presentation of a first user interface window, whichmay include a plurality of menus and selectors such as drop down menus,buttons, text boxes, hyperlinks, etc. associated with data analysisapplication 222 as understood by a person of skill in the art. Asfurther understood by a person of skill in the art, various operationsmay be performed in parallel, for example, using a plurality of threads.

A session may be established with controller device 104. A “session”includes user device 200, controller device 104 that is a controllernode, and a plurality of worker devices of worker system 106. Userdevice 200 accepts commands from a user and relays instructions tocontroller device 104. Controller device 104 establishes a communicationnetwork with the worker devices of worker system 106, sendinginstructions to the worker devices of worker system 106, collecting andaggregating the results of computations from the worker devices ofworker system 106, and communicating final results to user device 200.Controller device 104 may utilize itself as a worker device. The workerdevices of worker system 106 receive instructions from controller device104, store and process data, and send the results of computations backto controller device 104. Worker devices of worker system 106 may alsocommunicate with each other directly to accomplish a task.

In an operation 500, a first indicator may be received that indicatesthe input dataset. For example, the first indicator indicates a locationand/or a name of the input dataset. As an example, the first indicatormay be received by data analysis application 222 after selection from auser interface window or after entry by a user into a user interfacewindow. In an alternative embodiment, the input dataset may not beselectable. For example, a most recently created dataset may be usedautomatically.

In an operation 502, a load of the input dataset may be requested. Forexample, user device 200 may request that the input dataset be loadedinto a table that is ready for processing. In an alternative embodiment,the input dataset may already be ready for processing.

In an operation 504, a second indicator of the plurality of variablesincluded in the input dataset may be received in a response to therequest to load the input dataset. For example, a list of variable namesin the order they are read from a first row of the input dataset may bereceived. In an alternative embodiment, the second indicator may not bereceived. Instead, the user may already know the plurality of variablesincluded in the input dataset or may obtain the list using anothermethod.

In an operation 506, a third indicator may be received that indicates aplurality of variables v_(i) of the input dataset to analyze for eachobservation vector x_(i) read from a row of the input dataset. Forexample, the third indicator indicates a list of input variables toanalyze by name, column number, etc. The name may be matched to a columnheader included in the first row of the input dataset. Other variablesmay not be analyzed. In an alternative embodiment, the third indicatormay not be received. For example, all of the variables may be analyzedautomatically.

In an operation 508, a fourth indicator may be received that indicates aplurality of policy parameter values. The plurality of policy parametervalues is used to define how the plurality of variables v_(i) aregrouped. Each policy parameter value of the plurality of policyparameter values may have a predefined default value that may be usedwhen a user does not specify a value for the policy parameter using thefourth indicator. Each policy parameter value may be received using aseparate indicator. For illustration, Table I below includes theplurality of policy parameter values in accordance with an exampleembodiment:

TABLE I Policy parameter name Test Default value Very high-cardinality(high-C) threshold ≥ 5000 Cardinality ratio threshold for nominal > 0.25Non-integral values are interval flag = True Negative values areinterval flag = True Missing rate high threshold (%) ≥ 50 Missing ratemedium threshold (%) ≥ 5 Missing rate low threshold (%) < 5 Nominalcardinality high threshold ≥ 100 Nominal cardinality medium threshold ≥25 Nominal cardinality low threshold < 25 Nominal entropy high threshold(either Shannon ≥ 0.5, 0.5 entropy or Gini index) Nominal entropy mediumthreshold (either Shannon ≥ 0.1, 0.1 entropy or Gini index) Nominalentropy low threshold (both Shannon entropy < 0.1, 0.1 and Gini index)Nominal frequency skewness high threshold ≥ 100, 25, 0.7 (Top1/Bot1 orTop1/Top2 or variation ratio) Nominal frequency skewness mediumthreshold ≥ 1, 1, 0.5 (Top1/Bot1 or Top1/Top2 or variation ratio)Nominal frequency skewness low threshold < 1, 1, 0.5 (Top1/Bot1 andTop1/Top2 or variation ratio) Interval skew high threshold (classical oraverage >   10, 0.75 quantile) Interval skew medium threshold (classicalor average >   2, 0.1 quantile) Interval skew low threshold (classicaland average <   2, 0.1 quantile) Interval kurtosis high threshold(classical or average > 10, 3  quantile) Interval kurtosis mediumthreshold (classical or >   5, 2.75 average quantile) Interval kurtosislow threshold (classical and average <   5, 2.75 quantile) Intervaloutlier high threshold (%) ≥ 5 Interval outlier medium threshold (%) ≥ 1Interval outlier rate low threshold (%) < 1 Number of register bits Not10 applicable List of required nominal variables Not User applicableselected List of required interval variables Not User applicableselected Non-integral values interval = True Negative values interval =True

Referring to FIGS. 8A to 8C, user interface options provided by dataanalysis application 222 to allow the user to select the plurality ofpolicy parameter values are shown in accordance with an illustrativeembodiment. For example, FIG. 8A shows a first user interface window 800that allows the user to select a value for the very high-cardinality(high-C) threshold, a value for the cardinality ratio threshold for avariable to define the variable as a nominal variable, a checkbox todefine variables with non-integral values as interval variables, acheckbox to define variables with negative values as interval variables,a value for the medium missing rate threshold, and a value for the highmissing rate threshold. First user interface window 800 also includes anominal variable list box 802 and an interval variable list box 804.Required nominal variable list box 802 includes a list of each variableby name included in the input dataset or included in the plurality ofvariables v_(i) of the input dataset to analyze and defined in operation506. The user may select zero or more variables that are defined as thelist of required nominal variables regardless of any comparison with thedefined thresholds. Required interval variable list box 804 includes alist of each variable by name included in the input dataset or includedin the plurality of variables v_(i) of the input dataset to analyze anddefined in operation 506. The user may select zero or more variablesthat are defined as the list of required interval variables regardlessof any comparison with the defined thresholds.

FIG. 8B shows a second user interface window 806 that allows the user toselect a value for the nominal cardinality medium threshold, a value forthe nominal cardinality high threshold, a value for the nominal entropymedium threshold, a value for the nominal entropy high threshold, avalue for the nominal variation ratio high threshold, a value for thenominal variation ratio medium threshold, a value for the nominalvariation ratio low threshold, a value for the nominal frequency ratiomedium threshold Top1/Top2, a value for the nominal frequency ratio highthreshold Top1/Top2, a value for the nominal frequency ratio mediumthreshold Top1/Bot1, and a value for the nominal frequency ratio highthreshold Top1/Bot1.

FIG. 8C shows a third user interface window 808 that allows the user toselect a value for the interval classical skew medium threshold, a valuefor the interval classical skew high threshold, a value for the intervalrobust skew medium threshold, a value for the interval robust skew highthreshold, a value for the interval classical kurtosis medium threshold,a value for the interval classical kurtosis high threshold, a value forthe interval robust kurtosis medium threshold, a value for the intervalrobust kurtosis high threshold, a value for the interval outlierpercentage threshold medium, and a value for the interval outlierpercentage threshold high. A checkbox may indicate whether or not todetect a variable that has a variance value of zero.

The user interface options may initially be presented with the defaultvalues. In some cases, a low value for a policy parameter may not bespecified because it is identified as any variable value that is not“high” or “medium”. In an alternative embodiment, a high value for apolicy parameter may not be specified because it is identified as anyvariable value that is not “low” or “medium”. Though a hierarchy of low,medium, and high is used to group variables, a fewer or a greater numberof hierarchy levels may be specified to further categorize variables.

Referring again to FIG. 5, in an operation 510, a request to analyze theinput dataset based on the plurality of policy parameter values is sentto controller device 104. For example, the user may select a button toindicate that the plurality of policy parameter values has been selectedand that analysis of the input dataset should be performed. Theplurality of policy parameter values may be sent in a message or otherinstruction to controller device 104 or may be provided in a knownmemory location to controller device 104. In an alternative embodiment,user device 200 and controller device 104 may be integrated in the samecomputing device so that when the plurality of policy parameter valuesis received by user device 200, it is also received by controller device104.

In an operation 512, data analysis results are received. For example,variable statistical metrics and variable grouping data may be receivedfrom controller device 104 and stored in data analysis results 223 oncomputer-readable medium 208. The variable statistical metrics andvariable grouping data may be received from controller device 104. Asanother example, an indicator may be received that indicates that theanalysis process is complete and data analysis results 223 may alreadycontain the variable statistical metrics and variable grouping data. Forexample, one or more output tables may be presented on display 216 whenthe analysis process is complete. As another option, display 216 maypresent a statement indicating that the analysis process is complete.The user can access the variable statistical metrics and variablegrouping data in a predefined location or a user defined location ofdata analysis results 223.

In an operation 514, one or more results may be presented on display216. For example, the user may select a pair of policy parameter metricsto compare. For illustration, FIG. 9 shows a fourth user interfacewindow 900 that allows the user to select a first policy parameter usingan x-axis selector 902 to plot on an x-axis of a graph 906 and a secondpolicy parameter using a y-axis selector 904 to plot on a y-axis ofgraph 906. For the input dataset used to create the results shown inFIG. 9, controller data analysis application 312 identified 35 intervalvariables with the remaining 444 variables identified as nominalvariables. Whether interval or nominal variable groups are shown isbased on the policy parameter selections using x-axis selector 902 andy-axis selector 904. A table 908 summarizes a number of variables ineach hierarchical group based on the designation of high, medium, andlow for the selected x- and y-metrics though again a different number ofhierarchical groups may be defined. Each circle 910 shown in thescatterplot of graph 906 represents a pair of computed values for theselected policy parameter metrics selected using x-axis selector 902 andy-axis selector 904 for a variable of the plurality of variablesincluded in the input dataset. A first vertical line 912 indicates thethreshold between low and medium values for the x-axis metric. A secondvertical line 914 indicates the threshold between medium and high valuesfor the x-axis metric. A first horizontal line 916 indicates thethreshold between low and medium values for the y-axis metric. A secondhorizontal line 918 indicates the threshold between medium and highvalues for the y-axis metric.

For illustration, FIG. 10 shows a tree schematic 1000 that can bepresented on display 216 to summarize the groupings of the plurality ofvariables v_(i), where (#) is filled in with a number of variables thatsatisfy the specified grouping criteria. For example, a root node 1001of tree schematic 1000 indicates a number of the plurality of variablesv_(i). A nominal variable type node 1002 indicates a number of theplurality of variables v_(i) identified as a nominal variable based onthe plurality of policy parameter values. An interval variable type node1004 indicates a number of the plurality of variables v_(i) identifiedas an interval variable based on the plurality of policy parametervalues. A high-cardinality variable type node 1006 indicates a subset ofthe nominal variables identified as having a high-cardinality based onthe plurality of policy parameter values. A non-high-cardinalityvariable type node 1008 indicates a subset of the nominal variablesidentified as not having a high-cardinality based on the plurality ofpolicy parameter values.

A high-cardinality variable type table node 1010 summarizes the subsetof the high-cardinality variables identified as having a high, a medium,and a low missing rate based on the plurality of policy parametervalues. A non-high-cardinality variable type table node 1012 summarizesthe subset of the nominal variables identified as not havinghigh-cardinality. Each row of non-high-cardinality variable type tablenode 1012 defines the number of variables having the associatedcombination of high, medium, and low missing rate, cardinality, entropy,and frequency skewness based on the plurality of policy parametervalues. An interval variable type table node 1014 summarizes a number ofthe subset of the interval variables identified as having the associatedcombination of high, medium, and low missing rate, skewness, kurtosis,and outlier percentage based on the plurality of policy parametervalues.

Referring to FIG. 6, example operations associated with controller dataanalysis application 312 are described. Additional, fewer, or differentoperations may be performed depending on the embodiment. The order ofpresentation of the operations of FIG. 6 is not intended to be limiting.Again, controller data analysis application 312 and data analysisapplication 222 may be integrated or be the same applications so thatthe operations of FIGS. 5 and 6 are merged.

In an operation 600, the request to load the input dataset selected bythe user is received, if the input dataset is not already loaded.

In an operation 602, the input dataset is partitioned across each workerdevice 400 of worker system 106. After distributing the input dataset,input data subset 414 is stored in computer-readable medium 408 of eachworker device 400. In an alternative embodiment, the input dataset mayalready be loaded and distributed across each worker device 400.

In an operation 604, the analysis request may be received from userdevice 200 or directly from the user of user device 200 when integrated.

In an operation 606, the plurality of policy parameter values isextracted from the analysis request. In an alternative embodiment, therequest may include a reference to a location that is storing thevalues. In another alternative embodiment, the plurality of policyparameter values may be read from a known storage location.

In an operation 608, parameters are initialized. For example, controllerregister banks (hash tables) for each variable of the plurality ofvariables v_(i) are initialized to zero. The number of register bitspolicy parameter value may be used to define a size of the controllerregister banks according to the algorithm described in Stefan Heule etal., HyperLogLog in Practice: Algorithmic Engineering of a State of theArt Cardinality Estimation Algorithm, Proceedings of the 16thInternational Conference on Extending Database Technology, ACM, Mar. 18,2013, at 683 (HyperLogLog++). The HyperLogLog++ algorithm is a scalable,one-pass, approximate cardinality estimator used to estimate thecardinality and cardinality ratio statistics for each variable of theplurality of variables v_(i). Because users may include an analysis ofall variables in the input dataset, where there may be a large number ofvariables, a scalable execution of a first phase to estimate cardinalityis dependent on a scalability of the technique used, which excludesexact distinct count techniques that require an O(n) memory footprintthat cannot be applied to many modern datasets. This is especially truewhen the input dataset is a dataset with which the user is not familiarand has little, or no expert guidance on which variables can safely beexcluded from the predictive modelling workflow.

In an operation 610, computation of first phase statistics analysis ofthe input dataset distributed to worker system 106 may be requested ofeach worker device 400 of worker system 106. The first phase is used toclassify each variable of the plurality of variables v_(i) into nominaland interval variables and to split the nominal variables into twogroups. The first group has a very high cardinality based on exceedingthe very high-cardinality threshold policy parameter value. The secondgroup includes the nominal variables that do not exceed the veryhigh-cardinality threshold policy parameter value. The request mayinclude the number of register bits policy parameter value or eachworker device 400 may have access to the value for its computations.

In an operation 612, the first phase statistics analysis values may bereceived from each worker device 400 of worker system 106. For example,the register banks computed by each worker device 400 of worker system106 for each variable of the plurality of variables v_(i) may bereceived. Additionally, statistics such as a number of observations anda missing count value for each variable of the plurality of variablesv_(i) may be received. The first phase statistics analysis values may besent in a message or other instruction to controller device 104, may beprovided in a known memory location to controller device 104, returnedin a call to controller device 104, etc.

In an operation 614, an estimated cardinality value C_(e) is computedfrom the received first phase statistics analysis values from eachworker device 400 for each variable of the plurality of variables v_(i).For example, the register banks from each worker device 400 of workersystem 106 are processed iteratively to update controller register banksaccording to the HyperLogLog++ algorithm. A missing rate value M_(r)also may be computed for each variable of the plurality of variablesv_(i) by dividing a received missing count value from each worker device400 by the received number of observations from each worker device 400of the associated variable of the plurality of variables v_(i) such thatM_(r)=M/N, where N is the received number of observations and M is thereceived missing count value. A cardinality ratio value C_(r) also maybe computed for each variable of the plurality of variables v_(i) bydividing a computed cardinality value by the received number ofobservations minus the received missing count value of the associatedvariable of the plurality of variables v_(i) such thatC_(r)=C_(e)/(N−M).

Variables in a dataset are primarily either of numeric type or ofnon-numeric type. While non-numeric variables are always nominals,numeric variables can be either nominal or interval. For most practicalpredictive algorithms, the interval and nominal measurement scales arethe most important with others such as ordinal being subsumed by eitherscale.

In an operation 616, each variable of the plurality of variables v_(i)is determined to be an interval variable, a high-cardinality nominalvariable, or a non-high-cardinality nominal variable automatically basedon the plurality of policy parameter values. For example, any variableincluded in the required nominal variable list is identified and grouped(typed) as a nominal variable, and any variable included in the requiredinterval variable list is identified and grouped as an intervalvariable. Of the remaining non-grouped variables, variables that havenon-integral values or negative values are identified and grouped asinterval variables when the non-integral values interval policyparameter value or the negative values interval policy parameter value,respectively, are true.

Of the remaining non-grouped variables, the cardinality ratio valueC_(r) computed for each variable is compared to the cardinality ratiothreshold for nominal. Variables for which the cardinality ratio valueC_(r) greater than the cardinality ratio threshold for nominal specifiedby the plurality of policy parameter values are identified and grouped(typed) as a nominal variable.

For the variables identified as nominal variables, the cardinality valuecomputed for the associated variable is compared to the veryhigh-cardinality threshold policy parameter value. Those nominalvariables with cardinality values greater than the very high-cardinalitythreshold policy parameter value are identified and grouped ashigh-cardinality nominal variables. The remaining nominal variables areidentified and grouped as non-high-cardinality nominal variables.

For the variables identified as nominal variables, the cardinality valuecomputed for the associated variable is compared to the veryhigh-cardinality threshold policy parameter value. Those nominalvariables with cardinality values greater than the very high-cardinalitythreshold policy parameter value are identified and grouped ashigh-cardinality nominal variables. The remaining nominal variables areidentified and grouped as non-high-cardinality nominal variables.

For the high-cardinality nominal variables, the missing rate value M_(r)is compared to the missing rate high threshold M_(H), the missing ratemedium threshold M_(M), and the missing rate low threshold M_(L). Thehigh-cardinality nominal variables with M_(r)≥M_(H) are identified andgrouped as high-cardinality nominal variables with a high missing rate.The high-cardinality nominal variables with M_(r)≥M_(M) are identifiedand grouped as high-cardinality nominal variables with a medium missingrate. The high-cardinality nominal variables with M_(r)<M_(L) areidentified and grouped as high-cardinality nominal variables with a lowmissing rate.

As a result, after the first phase, each variable of the plurality ofvariables v_(i) is assigned to one of interval variable type node 1004,nominal high-cardinality variable type node 1006, or nominalnon-high-cardinality variable type node 1008. Each high-cardinalitynominal variable has also been assigned to a level of nominal,high-cardinality variable type table node 1010 based on the plurality ofpolicy parameter values.

In an operation 618, computation of second phase statistics analysis ofthe input dataset distributed to worker system 106 may be requested ofeach worker device 400 of worker system 106. The second phase is used tofurther classify each nominal non-high-cardinality variable into acombination of high, medium, or low missing rate, cardinality, entropy,and frequency skewness based on the plurality of policy parametervalues. The second phase is also used to further classify each intervalvariable into a combination of high, medium, or low missing rate,skewness, kurtosis, and outlier percentage based on the plurality ofpolicy parameter values. The request may include a nominal list of thenon-high-cardinality nominal variables and an interval list of theinterval variables or each worker device 400 may have access to thelists for its computations. The high-cardinality nominal variables areexcluded from the second phase analysis, which provides scalability sothat computation of the entropy and frequency skewness are not sloweddown by the high-cardinality nominal variables. Furthermore, thecardinality and missing rate values are sufficient to characterizehigh-cardinality nominal variables because these variables are commonlytransformed into interval scale for downstream analytics.

In an operation 620, second phase statistics values may be received fromeach worker device 400 of worker system 106 for each variable in thenominal list and each variable in the interval list. For example, thesecond phase statistics values include values for the parameters belowused to compute the grouping values for each variable. The second phasestatistics values may be sent in a message or other instruction tocontroller device 104, may be provided in a known memory location tocontroller device 104, returned in a call to controller device 104, etc.

In an operation 622, the grouping values for each variable in thenominal list and each variable in the interval list are computed fromthe received second phase statistics values for eachnon-high-cardinality variable and for each interval variable of theplurality of variables v_(i). The missing rate value M and thecardinality ratio value C_(r) were computed in operation 614.

For example, the Shannon entropy E_(S) may be computed for eachnon-high-cardinality nominal variable using

${E_{S} = \frac{- {\sum\limits_{i = 1}^{N_{N}}\; {p_{i}\log_{2}^{p_{i}}}}}{\log_{2}^{C}}},{{{where}\mspace{14mu} p_{i}} = \frac{f_{i}}{N_{N}}},$

where f_(i) is a number of times a unique value for the variableoccurred, N_(N) is a number of observations of the variable, and C isthe computed cardinality value defined based on a number of uniquevalues of the variable. The Gini entropy E_(G) may be computed for eachnon-high-cardinality nominal variable using

$E_{G} = {\frac{{- C}\mspace{14mu} {\sum\limits_{i = 1}^{N_{N}}\; ( {1 - p_{i}^{2}} )}}{( {C - 1} )}.}$

The variation ratio v may be computed for each non-high-cardinalitynominal variable using

${v = \frac{( {1 - f_{m}} )}{N_{N} - M}},$

where f_(m) is a frequency of a mode computed for the variable. A firstfrequency skewness F_(t1,t2) (Top1/Top2) may be computed for eachnon-high-cardinality nominal variable using F_(t1,t2)=f_(t1)/f_(t2),where f_(t1) is a frequency of a most frequent unique value and f_(t2)is a frequency of a second most frequent unique value. A secondfrequency skewness F_(t1,b1) (Top1/Bot1) may be computed for eachnon-high-cardinality nominal variable using F_(t1,b1)=f_(t1)/f_(b1),where f_(b1) is a frequency of a least frequent unique value. Eachnon-high-cardinality variable has a tuple of grouping values thatinclude the missing rate value M_(r), the cardinality ratio value C_(r),the Shannon entropy E_(S), the Gini entropy E_(G), the variation ratiov, the first frequency skewness F_(t1,t2), and the second frequencyskewness F_(t1,b1) or (M_(r), C_(r), E_(S), E_(G), v, F_(t1,t2),F_(t1,b1)). The Shannon entropy E_(S) and the Gini entropy E_(G) are acombined metric, and the variation ratio v, the first frequency skewnessF_(t1,t2), and the second frequency skewness F_(t1,b1) are a combinedmetric as indicated in Table I.

For example, the classical skewness S_(c) may be computed for eachinterval variable using S_(c)=E[(x−E[x]³], where E[x] is a mean valuefor the variable The average quantile skewness S_(q) may be computed foreach interval variable using S_(q)=(E[x]−q₂)/E[|x−q₂|], where q₂ is amedian value for the variable. The classical kurtosis K_(C) may becomputed for each interval variable using K_(C)=E[(x−E[x])⁴]. Theaverage quantile kurtosis K_(q) may be computed for each intervalvariable using K_(q)=(U_(A)−L_(A))/(U_(B)−L_(B)), whereU_(A/B)=∫_(1-a/b) ¹F⁻¹(X)dx, L_(A/B)=∫₀ ^(a/b)F⁻¹ (X)dx, and F⁻¹(X) isan inverse cumulative density function, a is a lower quantile, and b isan upper quantile used for the computation of a lower (L_(A), L_(B)) andan upper (U_(A), U_(B)) contribution. Illustrative values are a=0.025and b=0.25. The expressions for L_(A), L_(B), U_(A), and U_(B) can becast into sums for the upper and lower tail of the distribution of thevalues, which controller device 104 computes using the contributionscomputed by each worker device 400. The number of outliers N_(o) may beestimated using an adjusted boxplot as described in Mia Hubert and EllenVandervieren, An Adjusted Boxplot for Skewed Distributions, 52 Comput.Stat. Data Anal. 5186 (2008). The outlier percentage O_(p) is computedusing O_(p)=N_(o)/N_(I), where N_(I) is a number of non-missing valuesfor the interval variable. Each interval variable has a tuple ofgrouping values that include the missing rate value M_(r), the classicalskewness S_(c), the average quantile skewness S_(q), the classicalkurtosis K_(C), the average quantile kurtosis K_(q), and the outlierpercentage O_(p) or (M_(r), S_(c), S_(q), K_(C), K_(q), O_(p)). Theclassical skewness S_(c) and the average quantile skewness S_(q) are acombined metric, and the classical kurtosis K_(C) and the averagequantile kurtosis K_(q) are a combined metric as indicated in Table I.

Worker data analysis application 412 executed by each worker device 400computes the contributions to the statistical value while controllerdevice 104 receives the contributions from each worker device 400 andcomputes the final values for each statistical value. For example, inthe case of the mean value E[x], each worker device 400 sends theircontribution to both the sum of each variable and the number of usedobservations of each variable and controller device 104 aggregates thesecontributions and computes the actual value for the mean E[x]. Ofcourse, the higher order moments such as S_(c)=E[(x−E[x])³] need thecomputation of more factors than the sum such as the sums of x², x³,etc.

In an operation 624, an index is assigned to each variable group. Forexample, a variable group index may be defined for each row ofhigh-cardinality variable type table node 1010, each row ofnon-high-cardinality variable type table node 1012, and each row ofinterval variable type table node 1014. For illustration, an index ofone may be assigned to high cardinality variables with a high missingrate; an index of two may be assigned to high cardinality variables witha medium missing rate; an index of three may be assigned to highcardinality variables with a low missing rate; an index of four may beassigned to interval variables with a low missing rate, low skewness,low kurtosis, low outlier percentage; and so on.

In an operation 626, a group index is assigned to each variable of theplurality of variables v_(i). For example, each identifiedhigh-cardinality variable may be assigned the variable group index basedon the missing rate value M_(r) comparison described in operation 616.

The grouping values computed for each variable in the nominal list maybe compared to the nominal policy parameter(s) of the plurality ofpolicy parameters. For example, the missing rate value M_(r) is comparedto the missing rate high threshold M_(H), the missing rate mediumthreshold M_(M), and the missing rate low threshold M_(L) to assign thefirst dimension of the tuple (missing rate, cardinality ratio, theentropy compound metric, and the frequency skewness compound metric) aseither high, medium, or low. Similarly, the cardinality ratio valueC_(r) is compared to the nominal cardinality high threshold and thenominal cardinality low threshold to assign the second dimension of thetuple (missing rate, cardinality ratio, the entropy compound metric, andthe frequency skewness compound metric) as either high or low. TheShannon entropy E_(S) and the Gini entropy E_(G) are compared to theappropriate nominal entropy high threshold, the appropriate nominalentropy medium threshold, and the appropriate nominal entropy lowthreshold to assign the third dimension of the tuple (missing rate,cardinality ratio, the entropy compound metric, and the frequencyskewness compound metric) as either high, medium, or low based on thepolicy parameter test values. The first frequency skewness F_(t1,t2),the second frequency skewness F_(t1,b1), and the variation ratio v arecompared to the appropriate nominal frequency ratio high threshold, theappropriate nominal frequency ratio medium threshold, and theappropriate nominal frequency ratio low threshold to assign the fourthdimension of the tuple (missing rate, cardinality ratio, the entropycompound metric, and the frequency skewness compound metric) as eitherhigh, medium, or low based on the policy parameter test values.

The grouping values computed for each variable in the interval list maybe compared to the interval policy parameter(s) of the plurality ofpolicy parameters. For example, the missing rate value M_(r) is comparedto the missing rate high threshold M_(H), the missing rate mediumthreshold M_(M), and the missing rate low threshold M_(L) to assign thefirst dimension of the tuple (missing rate, the skewness compoundmetric, the kurtosis compound metric, outlier percentage) as eitherhigh, medium, or low. The classical skewness S_(c) and the averagequantile skewness S_(q) are compared to the appropriate interval skewhigh threshold, the appropriate interval skew medium threshold, and theappropriate interval skew low threshold to assign the second dimensionof the tuple (missing rate, the skewness compound metric, the kurtosiscompound metric, outlier percentage) as either high, medium, or lowbased on the policy parameter test values. The classical kurtosis K_(C)and the average quantile kurtosis K_(q) are compared to the appropriateinterval kurtosis high threshold, the appropriate interval kurtosismedium threshold, and the appropriate interval kurtosis low threshold toassign the third dimension of the tuple (missing rate, the skewnesscompound metric, the kurtosis compound metric, outlier percentage) aseither high, medium, or low based on the policy parameter test values.The outlier percentage O_(p) is compared to the interval outlier highthreshold, the interval outlier medium threshold, and the intervaloutlier low threshold to assign the fourth dimension of the tuple(missing rate, the skewness compound metric, the kurtosis compoundmetric, outlier percentage) as either high, medium, or low.

In an operation 628, data analysis results 223 may be returned orotherwise provided to user device 200 if user device 200 and controllerdevice 104 are not integrated. For example, the grouping values computedfor each variable may be returned with the assigned group index and/orthe assigned group of interval variable, non-high-cardinality nominalvariable, or high-cardinality nominal variable. In addition, a count ofa number of variables assigned to each variable group may be computed inoperation 626 as the group index is assigned to each variable and thecount of the number of variables assigned to each variable group may bereturned. A total number of variables, a total number of nominalvariables, a total number of interval variables, a number ofhigh-cardinality variables, and/or a number of non-high-cardinalityvariables also may be returned. A proportion of the number of variableswith a specific pair, triplet, etc. (any valid composition ofstatistical metrics based on the type of the variable) of statisticalmetrics also may be computed and returned. For example, a proportion ofinterval variables with a high missing rate metric and a low skewnessmetric may be computed and returned after operation 626.

Referring to FIGS. 7A and 7B, example operations associated with workerdata analysis application 412 are described. Additional, fewer, ordifferent operations may be performed depending on the embodiment. Theorder of presentation of the operations of FIGS. 7A and 7B is notintended to be limiting. Again, controller data analysis application 312and worker data analysis application 412 may be integrated or be thesame applications so that the operations of FIG. 6 and FIGS. 7A and 7Bare merged.

In an operation 700, a portion of the input dataset is received andloaded in third computer-readable medium 408 as input data subset 414unless already loaded.

In an operation 702, the request to compute first phase data values isreceived.

In an operation 704, the first phase data values are initialized. Forexample, worker register banks (hash tables) for each variable of theplurality of variables v_(i) are initialized to zero. The number ofregister bits policy parameter value that may have been received withthe request is used to define a size of the worker register banksaccording to the HyperLogLog++ algorithm. As another example, missingcounter values for each variable of the plurality of variables v_(i) areinitialized to zero. As yet another example, a number of observationscounter value is initialized to zero.

In an operation 706, a first observation is read from input data subset414 to define values for each variable of the plurality of variablesv_(i).

In an operation 708, the first phase data values are updated based onthe defined values. For example, the missing counter value isincremented for any variable for which a value is missing, the number ofobservations counter value is incremented, and the values used toestimate cardinality value C_(e) according to the HyperLogLog++algorithm are updated. Quantiles used for the average quantile skewnessand kurtosis metrics (a=0.025, b=0.25, and, q=0.5) may be computed aspart of the first phase data values for all numeric variables.

In an operation 710, a determination is made concerning whether inputdata subset 414 includes another observation. If input data subset 414includes another observation, processing continues in operation 706. Ifinput data subset 414 does not include another observation, processingcontinues in an operation 712.

In operation 712, the updated first phase data values computed for eachvariable of the plurality of variables v_(i) are returned or otherwiseprovided to controller device 104. The updated first phase data valuesmay be stored in subset statistics dataset 416.

In an operation 714, the request to compute second phase data values isreceived.

In an operation 716, the second phase data values are initialized. Forexample, frequency counter values, unique value counter values, sumvalues, and/or sum squared values for each variable of the plurality ofvariables v_(i) may be initialized to zero or one as appropriate.

In an operation 718, a first observation is read from input data subset414 to define values for each variable of the plurality of variablesv_(i).

In an operation 720, the second phase data values are updated based onthe defined values.

In an operation 722, a determination is made concerning whether inputdata subset 414 includes another observation. If input data subset 414includes another observation, processing continues in operation 718. Ifinput data subset 414 does not include another observation, processingcontinues in an operation 724.

In operation 724, the updated second phase data values computed for eachvariable of the plurality of variables v_(i) are returned or otherwiseprovided to controller device 104. The second first phase data valuesmay be stored in variable grouping data 418.

Quantifying data-quality issues of the input dataset is an importantfirst task in predictive modelling. Data analysis application 222,controller data analysis application 312, and worker data analysisapplication 412 need minimal inputs to organize the variables of theinput dataset into groups that are defined by statistical metrics. Thisorganization quantifies data quality issues of the dataset in an easilydigestible form. Thus, as a first pass, users can use data analysisapplication 222, controller data analysis application 312, and workerdata analysis application 412 with its default policy settings toaugment the normal data exploration part of their analytics workflow.Additionally, the user can easily adjust the policy parameter values.Though the default values are usually effective for most input datasets,it may be beneficial to experiment with different values for the policyparameters. This helps to identify variables that have borderline valuesfor specific statistical metrics. These variables can further beexplored individually for a better understanding and a more robustclassification. The graphical and numerical depiction of the results, asillustrated in FIGS. 9 and 10, is beneficial to understand the fullarray of data quality issues uncovered by data analysis application 222,controller data analysis application 312, and worker data analysisapplication 412. In the context of predictive modelling, the results canbe used as an input to a data transformation application 224 shownreferring to FIG. 11 or a high-C data transformation application 230shown referring to FIG. 17. For example, it is well-known that bothskewness reducing functional transformations such as Box-Coxtransformation and discretization can ameliorate skewness. However,skewness reducing functional transformations cannot handle issues due tomissing values. In contrast discretization can, as long as missingvalues are put in a distinct bin. Thus, both the treatment of missingvalues followed by skewness and the outright treatment of both missingvalues and skewness are potential treatments for a variable groupcharacterized by a high missing rate and a high skewness by datatransformation application 224.

Referring to FIG. 11, a second embodiment of user device 200 is shownthat further includes data transformation application 224. Datatransformation application 224 performs operations associated withrequesting transformation of the input dataset so that the user canbetter utilize the data in subsequent predictive model training. Theoperations may be implemented using hardware, firmware, software, or anycombination of these methods. Referring to the example embodiment ofFIG. 11, data transformation application 224 is implemented in software(comprised of computer-readable and/or computer-executable instructions)stored in computer-readable medium 208 and accessible by processor 210for execution of the instructions that embody the operations of datatransformation application 224. Data transformation application 224 maybe written using one or more programming languages, assembly languages,scripting languages, etc. Data transformation application 224 may beimplemented as a Web application.

Data transformation application 224 may be integrated with otheranalytic tools including data analysis application 222. As an example,data transformation application 224 may be part of an integrated dataanalytics software application and/or software architecture such as thatoffered by SAS Institute Inc. of Cary, N.C., USA. For example, datatransformation application 224 may be part of SAS® Enterprise Miner™developed and provided by SAS Institute Inc. of Cary, N.C., USA. Merelyfor further illustration, data transformation application 224 may beimplemented using or integrated with one or more SAS software tools suchas Base SAS, SAS/STAT®, SAS® High Performance Analytics Server, SAS®LASR™, SAS® In-Database Products, SAS® Scalable Performance Data Engine,SAS/OR®, SAS/ETS®, SAS® Inventory Optimization, SAS® InventoryOptimization Workbench, SAS® Visual Data Mining and Machine Learning,SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®,SAS® Forecast Server, all of which are developed and provided by SASInstitute Inc. of Cary, N.C., USA.

Referring to FIG. 12, a second embodiment of controller device 104 isshown that further includes controller data transformation application324. Controller data transformation application 324 performs operationsassociated with transforming the input dataset based on transformationflow parameter values 326 provided from user device 200 using thecomputing devices of worker system 106, when the input dataset isdistributed across the computing devices of worker system 106. Theoperations may be implemented using hardware, firmware, software, or anycombination of these methods. Referring to the example embodiment ofFIG. 12, controller data transformation application 324 is implementedin software (comprised of computer-readable and/or computer-executableinstructions) stored in second computer-readable medium 308 andaccessible by second processor 310 for execution of the instructionsthat embody the operations of controller data transformation application324. Controller data transformation application 324 may be written usingone or more programming languages, assembly languages, scriptinglanguages, etc. Controller data transformation application 324 may beimplemented as a Web application.

Controller data transformation application 324 may be integrated withother analytic tools including with controller data analysis application312. As an example, controller data transformation application 324 maybe part of an integrated data analytics software application and/orsoftware architecture such as that offered by SAS Institute Inc. ofCary, N.C., USA. For example, controller data transformation application324 may be part of SAS® Enterprise Miner™ developed and provided by SASInstitute Inc. of Cary, N.C., USA. Merely for further illustration,controller data transformation application 324 may be implemented usingor integrated with one or more SAS software tools such as Base SAS,SAS/STAT®, SAS® High Performance Analytics Server, SAS® LASR™, SAS®In-Database Products, SAS® Scalable Performance Data Engine, SAS/OR®,SAS/ETS®, SAS® Inventory Optimization, SAS® Inventory OptimizationWorkbench, SAS® Visual Data Mining and Machine Learning, SAS® VisualAnalytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®, SAS®Forecast Server, all of which are developed and provided by SASInstitute Inc. of Cary, N.C., USA.

Referring to FIG. 13, a second embodiment of worker device 400 is shownthat further includes worker data transformation application 424. Workerdata transformation application 424 performs data transformation ofinput data subset 414 based on inputs from controller device 104 todefine subset statistical data values 426 and transformed data subset428. Worker data transformation application 424 may be integrated withother analytic tools including worker data analysis application 412. Asan example, worker data transformation application 424 may be part of anintegrated data analytics software application and/or softwarearchitecture such as that offered by SAS Institute Inc. of Cary, N.C.,USA. For example, worker data transformation application 424 may be partof SAS® Enterprise Miner™ developed and provided by SAS Institute Inc.of Cary, N.C., USA. Merely for further illustration, worker datatransformation application 424 may be implemented using or integratedwith one or more SAS software tools such as Base SAS, SAS/STAT®, SAS®High Performance Analytics Server, SAS® LASR™, SAS® In-DatabaseProducts, SAS® Scalable Performance Data Engine, SAS/OR®, SAS/ETS®, SAS®Inventory Optimization, SAS® Inventory Optimization Workbench, SAS®Visual Data Mining and Machine Learning, SAS® Visual Analytics, SAS®Viya™, SAS In-Memory Statistics for Hadoop®, SAS® Forecast Server, allof which are developed and provided by SAS Institute Inc. of Cary, N.C.,USA.

Data transformation application 224, controller data transformationapplication 324, and worker data transformation application 424 may bethe same or different applications that are integrated in variousmanners to perform data transformation of the input dataset distributedacross worker system 106. Data transformation application 224,controller data transformation application 324, and worker datatransformation application 424 process a transformation request composedof user specifiable and configurable transformation flows. The user canspecify multiple, connected transformation phases per flow in a userconfigurable order. These transformation phases include imputation,outlier detection and treatment, functional transformation, anddiscretization phases for interval variable transformations, andimputation, map-interval, hashing, and nominal grouping phases fornominal variable transformations. Any one of these phases may beincluded or excluded in a particular transformation flow, and the phasescan be configured in a user-specifiable order. Data transformationapplication 224, controller data transformation application 324, andworker data transformation application 424 provide the capability tosuccinctly express most, if not all, feature transformations thatpractitioners apply in their predictive modeling workflow, therebyresulting in faster model development.

The composition of the most effective feature transformation stepsdepends on the particular modelling task, and in particular, the chosenpredictive model type. As a result, practitioners are forced toiteratively apply and evaluate feature transformation steps in theirpredictive modeling workflow. This makes feature transformation acombinatorial problem that requires the exploration of a large featuretransformation space. The user of data transformation application 224,controller data transformation application 324, and worker datatransformation application 424 can specify multiple featuretransformation flows that are processed in distributed-mode/parallelsharing data passes, which results in a significant reduction in anumber of data passes that may be required to transform data to a formconsumable by different predictive model types. This is an importantcontribution as it lets users efficiently explore and evaluate a largenumber of transformation flows. There is also no generation of temporary(intermediate) datasets, which is especially important in a big-dataand/or cloud environment where the computer memory is not available tosupport intermediate datasets.

Data transformation application 224, controller data transformationapplication 324, and worker data transformation application 424 providean effective solution for this combinatorial problem through theirexpressive, parallel, multi-flow feature transformation pipelines. Eachfeature transformation pipeline is user specifiable and configurable,thereby avoiding undue restrictions in the exploration of the featuretransformation space. In addition, their parallel and multi-flowcapabilities let the user explore multiple feature transformationpipelines for each variable or group of variables in parallel and in thesame data pass, without generating temporary datasets.

Each feature transformation flow is an independent task, and as such, avariable may be a member of multiple transformation flows in support ofdifferent predictive model types that have different featuretransformation needs. As a result, the multi-flow capability reduces thenumber of data passes (reads and writes) that are needed to prepare datafor multiple predictive model types, which saves computing cycles,memory accesses, network communications, etc. Again, this is especiallyimportant in a big-data and/or cloud environment.

Data transformation application 224, controller data transformationapplication 324, and worker data transformation application 424 optimizecomputations using transformation flow similarity and variable flowanalysis to avoid redundant intermediate computations acrosstransformation flows. The transformation similarity and variable flowanalysis techniques analyze the specified transformation flows todetermine and deactivate redundant intermediate computations. Redundantintermediate computations are grouped into sets and only a singlerepresentative from each set is designated as computable. The rest ofthe members of the set are designated as non-computable. Thenon-computable members share the result from their computablecounterpart avoiding redundant computations.

Referring to FIG. 14, example operations associated with datatransformation application 224 are described. Additional, fewer, ordifferent operations may be performed depending on the embodiment. Theorder of presentation of the operations of FIG. 14 is not intended to belimiting. A user can interact with one or more user interface windowspresented to the user in a display under control of data transformationapplication 224 independently or through a browser application in anorder selectable by the user. Although some of the operational flows arepresented in sequence, the various operations may be performed invarious repetitions, concurrently, and/or in other orders than thosethat are illustrated.

A session may be established with controller device 104.

Similar to operation 500, in an operation 1400, the first indicator maybe received that indicates the input dataset.

Similar to operation 502, in an operation 1402, a load of the inputdataset may be requested.

In an operation 1404, a fifth indicator may be received that indicates aplurality of transformation flow parameter values. The plurality oftransformation flow parameter values is used to define how each of aplurality of variables v_(i) are transformed though each variable can betransformed multiple times in different transformation flows. Eachtransformation flow parameter value of the plurality of transformationflow parameter values may have a predefined default value that may beused when a user does not specify a value for the transformation flowparameter using the fifth indicator.

In an operation 1406, a request to transform the input dataset based onthe plurality of transformation flow parameter values is sent tocontroller device 104. For example, the user may select a button toindicate that the plurality of transformation flow parameter values hasbeen selected and that transformation of the input dataset should beperformed. As another example, in the illustrative code above, the “run”statement triggers sending of the request to controller device 104. Theplurality of transformation flow parameter values may be sent in amessage or other instruction to controller device 104, may be providedin a known memory location to controller device 104, etc. In analternative embodiment, user device 200 and controller device 104 may beintegrated in the same computing device so that when the plurality oftransformation flow parameter values is received by user device 200, itis also received by controller device 104.

Each transformation flow parameter value may be received using aseparate indicator. For example, the following code establishes asession (“session mysess”) and sends a transformation request(“dataPreprocess.transform”) to process the input dataset defined bytable=“test”.

proc cas; session mysess; dataPreprocess.transform / table = ″test″requestPackages = { { name = ″pipeline1″ inputs = ${x1, x2} impute ={method = ″mean″} outlier = {method = ″IQR″, treatment=″trim″} function= {method = ″log″, args={otherArgs={10}}} discretize = {method =″bucket″} phaseOrder = “IFO” }, { name = ″pipeline2″ inputs = ${x1}impute = {method = ″mean″} function = {method = ″standardize″}discretize = {method = ″bucket″} }, { name = ″pipeline3″ inputs = ${x1,x2, x3} targets = ${y} outlier = {method = ″IQR″, treatment=″trim″}discretize = {method = ″MDLP″} },  { name = ″pipeline4″ inputs = ${c1,c2} impute = {method = ″mode″} catTrans = {method=″grouprare″,args={nbins=3}} }, { name = ″pipeline5″ inputs = ${c1} targets = ${y}events = {″1″} mapInterval = {method=″WOE″} } } casout = {name = ″out1″replace=True} ; run; quit;

The transformed dataset is stored in name=“out1”. The transformed valuesreplace the existing variable values though the option replace=Falseresults in the transformed values being added to the existing variablevalues instead of replacing them.

The “requestPackages” parameter defines a list of transformation flowsthat are the plurality of transformation flow parameter values thatdefine at least one transformation flow. Thus, each request package ofthe “requestPackages” parameter is a transformation flow definition ofone or more transformation flow definitions. Because each transformationflow can have multiple phases, computation of the parameters for a phaseis based on the data that flows from the preceding phase, if any. Forexample, if an interval transformation has an impute phase followed byfunctional transformation phase, the parameters of the functionaltransformation phase are estimated based on the imputedfeature/variable.

In the illustrative code, five transformation flow definitions named“pipeline1”, “pipeline2”, “pipeline3”, “pipeline4”, and “pipeline5” aredefined by the “requestPackages” parameter. The user can define anynumber of transformation flow definitions with each transformation flowdefinition associated with a transformation flow. Each transformationflow definition includes a “name” parameter that defines a name for thetransformation flow and an “inputs” parameter that defines a list of oneor more variables v_(tf,i) to be transformed by the transformation flowindicated by tf, where i=1, . . . , N_(tf) and N_(tf) is a number of theone or more variables listed for the transformation flow indicated bytf. For example, for the first transformation flow definition named“pipeline1”, N_(tf)=2, v_(1,1) is a variable named “x1” read from theinput dataset, and v_(1,2) is a variable named “x2” read from the inputdataset.

A transformation flow may include a “targets” parameter that defines alist of one or more target variables v_(tf,i) to be transformed by thetransformation flow indicated by tf, where i=1, . . . , N_(t,tf) andN_(t,tf) is a number of the one or more target variables listed for thetransformation flow indicated by tf. For example, for the thirdtransformation flow definition named “pipeline3”, N_(t,tf)=1, vt_(3,1)is a variable named “y” read from the input dataset.

For a binary target variable, an “events” parameter defines a targetvariable that the user has selected for modelling, such as a rare level.For example, for fraud detection with target variable y, if a value of“1” indicates fraud, then a value for the “events” parameter may be “1”.

Each transformation flow definition can be for either a nominaltransformation flow type or an interval transformation flow type basedon the type of variable(s) defined by the “inputs” parameter. Nominaltransformation flow types are transformations for which the inputvariables are nominal variables, for example, as identified by dataanalysis application 222, controller data analysis application 312, andworker data analysis application 412. Each nominal transformation flowcan include an impute phase (“impute”), a hash phase (“hash”), a mapinterval phase (“mapInterval”), and/or a categorical grouping phase(“catTrans”) that can be performed in the order they are defined in thetransformation flow. For example, the fourth transformation flowdefinition named “pipeline4” is an nominal transformation flow thatincludes an impute phase followed by a categorical grouping phase.

The impute phase for a nominal transformation flow type imputes a valuefor the specified input variables when a value is missing for anobservation using the specified method. The imputed value is a modestatistic computed for the variable and may be referred to as a phaseinternal parameter for the impute phase because it is computed prior toexecution of the transformation phase.

The hash phase maps values for the specified input variables using thespecified method.

The mapInterval phase maps values for the specified input variables toan interval scale using the specified method. As a result, these mappedvalues, essentially interval-scale intermediate variables, can befurther processed using an interval transformation. Level-value maps arethe phase internal parameters defined for the mapInterval phase. Eachlevel of the nominal variable is mapped to some interval/numeric valuedefined by the level-value maps. Optional methods for determining thelevel-value maps phase internal parameters include frequencies, eventprobabilities, weight of evidence (WOE), standardized centralizedmoments, etc.

The catTrans phase groups variables using the specified method.Level-group maps are the phase internal parameters defined for thecategorical grouping phase. Optional methods for determining thelevel-group maps phase internal parameters include unsupervised (rarelevel grouping “grouprare”) or supervised, such as decision tree,regression tree, etc., methods. Supervised methods use a target variablespecified by the “targets” parameter to perform the grouping. The “args”parameter defined for the catTrans phase varies dependent on the methodselected. For example, the rare level grouping, unsupervised method usesa number of bins into which the data is grouped. The number of bins isdefined by the “{nbins=3}” parameter.

Interval transformation flow types are transformations for which theinput variables are interval variables, for example, as identified bydata analysis application 222, controller data analysis application 312,and worker data analysis application 412. Each interval transformationflow type can include an impute phase (“impute”), an outlier phase(“outlier”), a functional transform phase (“function”), and/or adiscretize phase (“discretize”) that can be performed in the order theyare defined in the transformation flow definition. For example, thefirst transformation flow definition named “pipeline1” is an intervaltransformation flow type that includes an impute phase followed by anoutlier phase followed by a functional transform phase followed by adiscretize phase.

The impute phase for an interval transformation flow type imputes avalue for the specified input variables when a value is missing for anobservation using the specified method. Again, the imputed value is thephase internal parameter for the impute phase because it is computedprior to execution of the transformation phase. The imputed value may bea central tendency statistic computed for the variable that may be amean, a median, a Winsorized mean, a trimmed mean, a mid-range, ageometric mean, a harmonic mean, Tukey's biweight, etc. as understood bya person of skill in the art.

The outlier phase detects, using the specified method, and treats, usingthe specified treatment method, outlier values for the specified inputvariables. A lower threshold, an upper threshold, and a replacementvalue are the phase internal parameters defined for the outlier phase.Optional methods for computing the lower and upper threshold phaseinternal parameters include z-score, robust z-score, inter-quantilerange (IQR), percentile, user-defined limits, etc. Optional treatmentmethods for computing the replacement value phase internal parameterinclude winsorization, trimming and value replacement, etc. The outlierphase internal parameters depend on location and scale estimates.Location estimates may be computed as a mean, a median, a winsorizedmean, a trimmed mean, a mid-range, a geometric mean, a harmonic mean,Tukey's biweight, etc. Scale estimates may be computed as a standarddeviation, an IQR, a median absolute deviation about the median (MAD), aGini scale, a Tukey's biweight, etc. These statistics are used tocompute the phase internal parameter for the outlier phase depending onthe selected outlier detection and treatment methods.

The function phase transforms the specified input variables using thespecified method. Optional methods include log, sqrt, centering,standardization, etc. The “args” parameter defined for the functionphase varies dependent on the method selected. Depending on the methodselected the function phase may not need computation of any phaseinternal parameters. For example, the methods “log” and “sqrt” do notrequire any computation to perform the requested transformation. Datadependent methods such as centering and standardization include locationand scale estimates as phase internal parameters for the function phase.Location estimates may be computed as a mean, a median, a winsorizedmean, a trimmed mean, a mid-range, a geometric mean, a harmonic mean,Tukey's biweight, etc. Scale estimates may be computed as a standarddeviation, an IQR, a MAD, a Gini scale, a Tukey's biweight, etc. Thesestatistics are used to compute the phase internal parameter for datadependent methods selected for the function phase.

The discretize phase transforms the specified input variables using thespecified method. Cut-points (bin boundaries) are the phase internalparameters defined for the discretize phase. Computation of thecut-points depends on the specified discretization method. Thediscretization methods include non-iterative (unsupervised) anditerative (supervised) techniques. Non-iterative (unsupervised) methodssuch as bucket and equal-frequency compute the cut-points based onstatistics such as a minimum and a maximum or quantiles, and thespecified number of bins. In contrast, iterative (supervised) techniquessuch as a minimum description length principle (MDLP),extended-chi-merge, class-attribute contingency coefficient (CACC), etc.use statistics for construction of a contingency (frequency) table, andthe contingency table is processed by the specified method to estimatethe cut-points. The minimum description length principle and theextended-chi-merge technique is described in J. Dougherty et al.,Supervised and Unsupervised Discretization of Continuous Features,Proceedings 12th International Conference on Machine Learning, at 194(1995). The class-attribute contingency coefficient technique isdescribed in Cheng-Jung Tsai, Chien-I Lee, Wei-Pang Yang: Adiscretization algorithm based on Class-Attribute ContingencyCoefficient. Inf. Sci. 178(3): 714-731 (2008).

As stated previously, transformation flows/pipelines are of eitherinterval or nominal type. By default, if used, interval transformationphases are processed according to the following sequence: 1) imputephase, 2) outlier phase, 3) function phase, and 4) discretize phase. Bydefault, if used, nominal transformation phases are processed accordingto the following sequence: 1) impute phase, 2) hash phase, and 3)catTrans phase or 4) mapInterval phase. For interval transformationphases, the default phase order can be changed using the “phaseOrder”parameter. For example, setting the “phaseOrder” parameter value to“FOI” indicates the following sequence: 1) function (“F”) phase, 2)outlier (“O”) phase, 3) impute (“I”) phase, and 4) discretize phase. The“phaseOrder” parameter value does not affect the discretize phase, whichis applied last. Thus, the phase order for application of the function(“F”) phase, the outlier (“O”) phase, and the impute (“I”) phase can bechanged from the default order using the “phaseOrder” parameter valueand defining the order using the appropriate letter designation.

The user may use data analysis results 223 to identify the phases andorder of application of the phases to apply to specific variables. Forillustration, Table II below includes the plurality of transformationflow parameter values that can be selected by a user to define atransformation flow.

TABLE II Transformation flow parameter name Options Default values namename value none inputs list of one more variable none names imputemethod - for interval Mean for interval and transformation, mean, modefor nominal median, min, max, harmonic mean, winsorized mean, trimmedmean, geometric mean, user-provided value; for nominal, mode and user-provided value. outlier method - IQR, Z-score, IQR modified Z-score,trim percentile, and user defined limits. treatment - trim, winsor,replace function method - log, BoxCox, standardize exp, sqrt, power,standardize, center, . . . Options for location and scale estimatesdiscretize method - bucket, quantile, bucket MDLP, CACC, Chimerge,regressionTree (single predictor), WOE. Options to control the number ofbins (nbins, max nbins, min nbins). phaseOrder IOF, IFO, OIF, OFI, FIO,IOF FOI targets list of one more target none variable names catTransmethod - grouprare, grouprare WOE, decisionTree. Options to control thenumber of bins (nbins, max nbins, min nbins). events list of one or moreevents none for the binary target variables. mapInterval Method - WOE,event- none probability, standardized moments, counts

In the illustrative code above, the first, second, and thirdtransformation flows are interval transformation flows, while the fourthand fifth transformation flows are nominal transformation flows. Acomplexity of a transformation flow is expressed by an order value thatcounts a number of phases in the transformation flow where the order ofa transformation flow is a number of phases of the transformation flow.

In the illustrative code above, the first transformation flow is a4th-order transformation flow (impute phase, outlier phase, functionaltransform phase, discretize phase), the second transformation flow is a3rd-order transformation flow (impute phase, functional transform phase,discretize phase), the third transformation flow is a 2nd-ordertransformation flow (outlier phase, discretize phase), the fourthtransformation flow is a 2nd-order transformation flow (impute phase,catTrans phase), and the fifth transformation flow is a 1st-ordertransformation flow (mapInterval phase).

As mentioned above, various statistical values may need to be computedas part of execution of a phase and are referred to as phase internalparameters. Table III lists the statistics used to compute thetransformed values for each type of phase. The hash phase does notrequire any phase internal parameters.

TABLE III Statistic type impute outlier function discretize mapIntervalcatTrans Basic (nobs, Yes Yes Yes Yes Yes Yes number missingobservations, min, max) Location Yes Yes Yes estimate (mean, median,trimmed mean, winsorized mean, harmonic mean, geometric mean) Scaleestimate Yes Yes Yes (Std, IQR, MAD and Gini scale) Quantile Yes Yes YesYes Contingency Yes Yes Yes table Distinct counts Yes Yes Yes WOE,moments, Yes event probability, level frequency

Basic statistics include a number of observations, a minimum value, amaximum value, etc. Again, location estimates may include a mean, amedian, a winsorized mean, a trimmed mean, a mid-range, a geometricmean, a harmonic mean, Tukey's biweight, etc. Scale estimates mayinclude a standard deviation, an IQR, a MAD, a Gini scale, a Tukey'sbiweight, etc.

In an operation 1408, a status indicator may be received that indicatesa success or a failure of the transformation request. Additionally, orin the alternative, a summary table may be received that provides a listof transformed variables.

In an operation 1410, the received status indicator may be presented ondisplay 216.

Referring to FIGS. 15A and 15B, example operations associated withcontroller data transformation application 324 are described.Additional, fewer, or different operations may be performed depending onthe embodiment. The order of presentation of the operations of FIGS. 15Aand 15B is not intended to be limiting. Again, controller datatransformation application 324 and data transformation application 224may be integrated or be the same applications so that the operations ofFIG. 14 and FIGS. 15A and 15B are merged. Additionally, or in thealternative, controller data analysis application 312 and controllerdata transformation application 324 may be integrated or be the sameapplications so that the operations of FIG. 6 and FIGS. 15A and 15B aremerged.

Similar to operation 600, in an operation 1500, the request to load theinput dataset selected by the user is received.

Similar to operation 602, in an operation 1502, the input dataset ispartitioned across each worker device 400 of worker system 106.

In an operation 1504, the transformation request may be received fromuser device 200 or directly from the user of user device 200 whenintegrated.

In an operation 1506, the plurality of transformation flow parametervalues is extracted from the transformation request. In an alternativeembodiment, the request may include a reference to a location that isstoring the values. In another alternative embodiment, the plurality oftransformation flow parameter values may be read from a known storagelocation. The plurality of transformation flow parameter values may bestored in transformation flow parameter values 326.

In an operation 1508, an index value for tf is assigned to eachtransformation flow. For example, an index of one, tf=1, is assigned tothe first transformation flow; an index of two, tf=2, is assigned to thesecond transformation flow; an index of three, tf=3, is assigned to thethird transformation flow; an index of four, tf=4, is assigned to thefourth transformation flow; an index of five, tf=5, is assigned to thefifth transformation flow; etc.

In an operation 1510, a flow similarity between the plurality oftransformation flows defined by the plurality of transformation flowparameter values is determined. A transformation request can include alarge number of transformation flows. As a result, direct (naïve)computation of the statistics that are required to define the phaseinternal parameters for each phase may introduce significantinefficiencies due to redundant computations. This can be a performancebottleneck, especially in a big data or a distributed data environment.To avoid these inefficiencies, the similarity is determined to avoidredundant intermediate computations. The flow similarity analysis isdone for each phase of the transformation flows. Two transformationflows are n^(th)-order similar if the first n phases are similar. Flowsimilarity analysis is not based on the input variables and/or targetvariables specified for each transformation flow. Flow similarity ismeasured between transformation flows of the same type. For example,flow similarity is determined separately for interval transformationflows and for nominal transformation flows. For illustration,considering the code above, the first transformation flow is1^(st)-order similar with the second transformation flow and vice versabecause flow similarity is symmetric. The third transformation flow is0^(th)-order similar with the first transformation flow and with thesecond transformation flow.

For illustration, each entry in a similarity matrix SA[n, n] isinitialized to zero, where n is a number of the transformation flows.

for i = 1 to n for j = i + 1 to n for k = 1 to np, where np is a numberof phases of TF_(j), where TF_(j) is the j^(th) transformation flow Ifthe kth phase of TF_(i), where TF_(i) is the i^(th) transformation flow,is equivalent to the kth phase of TF_(j), SA[i, j]+= 1 end for end forend for

The equivalence of two phases is dependent on the full array of optionsspecified for those options. For example, if two phases are bothimputations, but one is a mean imputation and the other is medianimputation, the two phases are not equivalent. SA[i,j] holds thesimilarity order of i^(th) transformation flow with the j^(th)transformation flow, where the similarity order of a transformation flowwith itself is not computed and the similarity matrix is symmetric.

In an operation 1512, a maximum transformation order M_(to) isdetermined. For example, a maximum order is identified from the order ofeach transformation flow. For illustration, considering the code above,the maximum transformation order is four because the firsttransformation flow has four phases, which is the maximum order for anyof the five defined transformation flows.

In an operation 1514, a current order a is initialized to one.

In an operation 1516, a set of statistical computations is defined as ana^(th) order computation set. The set includes one or more tuplesdefined by (a, tf, v, st_type), where a is the current order selected inoperation 1514, tf is the transformation flow index assigned inoperation 1508 for the associated transformation flow, v is a variableof the associated transformation flow, and st_type is a statistic type.For illustration, considering the code above, Table IV captures thestatistic(s), if any, for each order and each transformation flow.

TABLE IV TF1 TF2 TF3 TF4 TF5 1^(st)-order mean mean quantile mode levelfrequency, WOE 2^(nd)-order quantile mean, std. min, max level dev.frequency 3^(rd)-order min, max 4^(th)-order min, max

The statistic(s) are determined based on the method(s) and phase(s)defined by the plurality of transformation flows and the phase internalparameters associated with each. For example, the third phase for thefirst transformation flow is a log function phase that does not includeany phase internal parameters. The set of statistical computationsdefined for the 1^(st) statistical computation set for a first iterationof operation 1516 includes (1, 1, x1, mean), (1, 1, x2, mean), (1, 2,x1, mean), (1, 3, x1, quantile), (1, 3, x2, quantile), (1, 3, x3,quantile), (1, 4, c1, mode), (1, 4, c2, mode), (1, 5, c1, levelfrequency), (1, 5, c1, WOE).

The set of statistical computations defined for the 2^(nd) statisticalcomputation set for a second iteration of operation 1516 includes (2, 1,x1, quantile), (2, 1, x2, quantile), (2, 2, x1, mean), (2, 2, x1, std.dev.), (2, 3, x1, min), (2, 3, x1, max), (2, 3, x2, min), (2, 3, x2,max), (2, 3, x3, min), (2, 3, x3, max), (2, 4, c1, level frequency), (2,4, c2, level frequency).

The set of statistical computations defined for the 3^(rd) statisticalcomputation set for a third iteration of operation 1516 includes (3, 2,x1, min), (3, 2, x1, max).

The set of statistical computations defined for the 4th statisticalcomputation set for a fourth iteration of operation 1516 includes (4, 1,x1, min), (4, 1, x1, max), (4, 1, x2, min), (4, 1, x2, max).

The set of statistical computations is created by looping through eachtransformation flow that has a remaining phase based on the order index,then looping through the input variables specified for the remainingphase, and then looping through the statistical parameters required forthe phase and method specified for the phase, if any.

In an operation 1518, a statistical computation index is assigned toeach statistical computation of the set of statistical computations. Forexample, for a first iteration of operation 1518, a statisticalcomputation index of one is assigned to (1, 1, x1, mean); a statisticalcomputation index of two is assigned to (1, 1, x2, mean); a statisticalcomputation index of three is assigned to (1, 2, x1, mean); astatistical computation index of four is assigned to (1, 3, x1,quantile); a statistical computation index of five is assigned to (1, 3,x2, quantile); a statistical computation index of six is assigned to (1,3, x3, quantile); a statistical computation index of seven is assignedto (1, 4, c1, mode); a statistical computation index of eight isassigned to (1, 4, c2, mode); a statistical computation index of nine isassigned to (1, 5, c1, level frequency); and a statistical computationindex of ten is assigned to (1, 5, c1, WOE) for a=1 and tf=1, . . . , 5.

In an operation 1520, any identical statistical computations withoutconsidering the transformation flow index tf are grouped. Statisticalcomputations are identical if their reduced tuples are identical. Forexample, the reduced tuples include (a, v, st_type), where a is thecurrent order selected in operation 1514, v is a variable of theassociated transformation flow, and st_type is a statistic type.

In an operation 1522, a statistical computation is selected from eachgrouped set of statistical computations.

In an operation 1524, the selected statistical computation from eachgrouped set of statistical computations is designated as active.

In an operation 1526, any remaining statistical computation(s) of eachgrouped set of statistical computations are designated as inactive.

In an operation 1528, a set of statistical computations is defined thatincludes any non-grouped statistical computation and the selectedstatistical computation designated as active for each grouped set ofstatistical computations. The inactive statistical computation(s) ofeach grouped set of statistical computations will receive the datacomputed for the corresponding statistical computation indicated asactive, but the value will not be redundantly computed.

In an operation 1530, a determination is made concerning whether thereis another order for which to define a computable set of statisticalcomputations. For example, when a=M_(to), there is not another order.When there is another order, processing continues in an operation 1532.When there is not another order, processing continues in an operation1534.

In operation 1532, the current order a is incremented by one. Forexample, a=a+1 and processing continues in operation 1516.

Shown referring to FIG. 15B, in operation 1534, the current order a isre-initialized to one.

In an operation 1536, a request is sent to each worker device 400 tocompute each statistical computation of the a^(th) computable set ofstatistical computations. For example, the request includes the tuplesassociated with each statistical computation included in the set toinstruct each worker device 400 to compute a specific statistic type fora specific variable and associate it with the order and transformationflow defined by the tuple.

In an operation 1538, the statistical results for each statisticalcomputation of the a^(th) computable set of statistical computations arereceived from each worker device 400.

In an operation 1540, a phase internal parameter value is computed foreach statistical computation for the current order. For example, themean of variable “x1” is computed for (1, 1, x1, mean) using thestatistical results for (1, 1, x1, mean) that include a counter of anumber of observations of “x1” and a sum of all of the observationvalues of “x1”.

In an operation 1542, the computed phase internal parameter value foreach statistical computation for the current order may be stored inphase internal parameter values dataset 328 with its associated tupleinformation.

In an operation 1544, a determination is made concerning whether thereis another order for which to compute the phase internal parametervalues. For example, when a=M_(to), there is not another order. Whenthere is another order, processing continues in an operation 1546. Whenthere is not another order, processing continues in an operation 1548.

In operation 1546, the current order a is incremented by one. Forexample, a=a+1, and processing continues in operation 1536.

In operation 1548, a final computation of the phase internal parametervalues is performed and may also be stored in phase internal parametervalues dataset 328 with its associated tuple information. No furthercomputation is required for some of the phase internal parameter valuessuch as a mean, a median, etc. However, additional computation is neededto compute some of the phase internal parameter values. For example, alower threshold and an upper threshold may be estimated from quantileestimates computed in operation 1540 using the specified method such asthe IQR formula. As another example, the bin boundaries or cut-pointsfor a discretize phase may be computed from minimum and maximumestimates using a contingency table. The contingency table is afrequency table that counts a number of occurrences of values of x (thetransformation variable) and y (the target variable—if specified). Thecontingency table is defined based on the type of discretize phase:

-   -   For x, if bucket binning or initialization is selected for the        supervised discretize phase, the cut-points of the contingency        table are generated using        cut-point(i)=min(x)+i*(min(x)−max(x))/m, where m is a number of        rows of the contingency table that is equal to a number of        splits of the x variable. For quantile binning or        initialization, the cut-points are set equal to the        corresponding quantiles.    -   For y, if specified, a number of unique values of y is        determined and the unique values are used to define the columns.        If y is not specified, the number of unique values is one.    -   During the data pass, values of x and y (in each record) are        used to map the observation to one of the cells of the        contingency table, and the frequency count of the mapped cell is        incremented by one.

The contingency table is a final output for unsupervised discretizephases (e.g. bucket/equal-width, quantile/equal-frequency). In contrast,the contingency table is processed further to generate the final binsfor supervised discretize phases such as MDLP, extended chi-merge, etc.

In an operation 1550, a request is sent to each worker device 400 totransform each variable for each transformation flow. For example, therequest includes the computed phase internal parameter value for eachstatistical computation with its associated tuple to instruct workerdevice 400 to perform each transformation associated with each phase foreach variable with the phase internal parameter value(s) needed by theassociated phase.

In an operation 1552, a done indicator is received from each workerdevice 400.

In operation 1554, a done indicator is sent to user device 200. Thetransformed data may be stored in transformed data subset 428 at eachworker device 400.

Referring to FIGS. 16A, 16B, and 16C, example operations associated withworker data transformation application 424 are described. Additional,fewer, or different operations may be performed depending on theembodiment. The order of presentation of the operations of FIGS. 16A,16B, and 16C is not intended to be limiting. Controller datatransformation application 324 and worker data transformationapplication 424 may be integrated or be the same applications so thatthe operations of FIGS. 15A and 15B and FIGS. 16A, 16B, and 16C aremerged. Additionally, or in the alternative, worker data analysisapplication 412 and worker data transformation application 424 may beintegrated or be the same applications so that the operations of FIGS.7A and 7B and FIGS. 16A, 16B, and 16C are merged.

Similar to operation 700, in an operation 1600, a portion of the inputdataset is received and loaded in third computer-readable medium 408 asinput data subset 414.

In an operation 1602, the request to compute a statistical value isreceived. The request may include the active set of statisticalcomputations for a current order being processing by controller device104. The request may further include the set of statistical computationsthat includes inactive statistical computations. For illustration, forthe first order provided in the example above, worker device 400receives (1, 1, x1, mean); (1, 1, x2, mean); (1, 2, x1, mean); (1, 3,x1, quantile); (1, 3, x2, quantile); (1, 3, x3, quantile); (1, 4, c1,mode); (1, 4, c2, mode); (1, 5, c1, level frequency); and (1, 5, c1,WOE), but, as in operations 1520 to 1528, identifies (1, 2, x1, mean) asinactive relative to (1, 1, x1, mean) because the computations areidentical except for the transformation flow.

In an operation 1604, statistical data values are initialized as neededfor each statistical computation. For example, counters and sum valuesare initialized to zero. Minimum values may be initialized to a largequantity, and maximum values may be initialized to a large negativequantity. Illustrative counters include a number of observations countervalue, a number of missing observations counter value, a number ofunique values counter value, a number of occurrences of each uniquevalue counter value, etc. for each statistical computation based on thetype of statistical computation. Illustrative sum values include a totalsum of values of each variable, a total sum of squared values of eachvariable, a total sum of inverse values of each variable, a total sum ofdifference values of each variable, etc. for each statisticalcomputation based on the type of statistical computation.

In an operation 1606, a first observation is read from input data subset414 to define values for each variable of the plurality of variablesv_(i).

In an operation 1608, the statistical data value(s) associated with eachstatistical computation are updated based on the defined values. Forexample, the missing counter value is incremented for any variable forwhich a value is missing; the number of observations counter value isincremented, a sum of values is updated, etc.

In an operation 1610, a determination is made concerning whether inputdata subset 414 includes another observation. If input data subset 414includes another observation, processing continues in operation 1606. Ifinput data subset 414 does not include another observation, processingcontinues in an operation 1612.

In operation 1612, the updated statistical data value(s) associated witheach statistical computation are returned or otherwise provided tocontroller device 104. The updated statistical data value(s) may bestored in subset statistical data values 426.

Referring to FIG. 16B, in an operation 1614, the request to transformeach variable of each transformation flow is received. For example, thereceived request includes the computed phase internal parameter valuefor each statistical computation with its associated tuple. The computedphase internal parameter value for each statistical computation with itsassociated tuple may be stored in subset statistical data values 426.

In an operation 1616, a first observation is read as a currentobservation from input data subset 414 to define values for eachvariable of the plurality of variables v_(i). Transformed data subset428 may be opened for writing on a first row. A current row oftransformed data subset 428 is the first row. When the optionreplace=False is selected by the user, the first observation may bewritten to transformed data subset 428 so that transformed values areappended to the original values read from input data subset 414. Aheader row may be written to the first row of transformed data subset428 that includes a variable name for each transformed variable. Forexample, the variable name for each variable to transform may beappended to the “name” parameter value given to each transformation flowso that each transformed variable has a unique name. For example, in theillustrative code above, a first transformed variable may be named“pipeline1_x1”, a second transformed variable may be named“pipeline1_x2”, a third transformed variable may be named“pipeline2_x1”, . . . , and a ninth transformed variable may be named“pipeline5_c1”.

In an operation 1618, a current transformation flow is initialized toone, tf=1.

In an operation 1620, a variable is selected as a current variable fromthe current transformation flow, and a current value V_(c) is defined asthe value for the current variable selected from the currentobservation. For example, for the first transformation flow provided inthe example code, the variable “x1” is selected as the current variable,and the current value is defined as the value of the variable “x1” ofthe current observation.

In an operation 1622, a current order (phase) is initialized to one,a=1.

In an operation 1624, a transformation function is defined for thecurrent order, the current transformation flow, and the current variableusing an identifier of the phase (e.g., “impute”, “function”,“discretize”) associated with the current order and the computed phaseinternal parameter value(s) associated with the current order, thecurrent transformation flow, and the current variable. For example, thephase is matched to a function call and passed the parameters and theread value.

In an operation 1626, a result variable value V_(r) is computed from thecurrent value using the defined transformation function. For example, ifthe current value indicates that a value for the current variableselected from the current observation is missing and the phase is imputewith a mean value, the defined transformation function selects the meanvalue provided as the computed phase internal parameter value(s)associated with the current order, the current transformation flow, andthe current variable and sets the result variable value equal to themean value. As another example, if the current phase is function(log),the defined transformation function computes a log of the current valueand sets the result variable value equal to that log value. As anotherexample, if the current phase is discretize(bucket), the definedtransformation function determines in which bin of the contingency table(provided as the computed phase internal parameter value(s) associatedwith the current order) the current value falls and sets the resultvariable value equal to that bin value.

In an operation 1628, a determination is made concerning whether or notthere is another order or phase of the current transformation flow toprocess. For example, when a=M_(tfo), there is not another order, whereM_(tfo) is a maximum order (number of phases) of the currenttransformation flow tf. When there is another order, processingcontinues in an operation 1630. When there is not another order,processing continues in an operation 1632.

In operation 1630, the current order a is incremented by one to point tothe next phase of the transformation flow and the current value is setequal to the computed result variable value V_(c)=V_(r). For example,a=a+1, V_(c)=V_(r), and processing continues in operation 1624 to applythe next phase to the result variable value of the previous phase.

Referring to FIG. 16C, in an operation 1632, the computed resultvariable value is appended to the current row of transformed data subset428 as an output value of the current transformation flow for the valueof the current variable.

In an operation 1634, a determination is made concerning whether thereis another variable to process for the current transformation flow toprocess. When there is another variable, processing continues in anoperation 1636. When there is not another variable, processing continuesin an operation 1638.

In operation 1636, a next variable is selected as the current variablefrom the current transformation flow, a current value V_(c) is definedas the value for the next variable selected from the currentobservation, and processing continues in operation 1622 to apply thecurrent transformation flow to the next variable. For example, for thefirst transformation flow provided in the example code, the variable“x2” is selected as the next variable, and the current value is definedas the value of the variable “x2” of the current observation.

In an operation 1638, a determination is made concerning whether thereis another transformation flow to process. When there is anothertransformation flow, processing continues in an operation 1640. Whenthere is not another transformation flow, processing continues in anoperation 1642.

In operation 1640, a next transformation flow is selected as the currenttransformation flow, and processing continues in operation 1620 to applythe next transformation flow. For example, because indices were assignedto each transformation flow, the transformation flow may be incrementedby one to index to the next transformation flow, tf=tf+1. Forillustration, after processing the first transformation flow, the secondtransformation flow is selected as the current transformation flow.

In an operation 1642, a determination is made concerning whether thereis another observation to process in input data subset 414 to definevalues for each variable of the plurality of variables v_(i). When thereis another observation, processing continues in an operation 1644. Whenthere is not another observation, processing continues in an operation1646.

In operation 1644, a next observation is read from input data subset414, and processing continues in operation 1618 to process the nextobservation. When the option replace=False is selected by the user, thenext observation may be written to transformed data subset 428 so thatthe transformed values are appended to the original values read frominput data subset 414 on a next row of transformed data subset 428. Whenthe option replace=True is selected by the user, the next observationmay not be written to transformed data subset 428 so that only thetransformed values are written to the next row of transformed datasubset 428. The current row of transformed data subset 428 is the nextrow.

In operation 1646, a done indicator is sent to controller device 104.

Predictive modelling practitioners such as data scientists andstatisticians, spend a significant part of their time in the datapreprocessing (feature transformation and generation) phase. Datatransformation application 224, controller data transformationapplication 324, and worker data transformation application 424transform the input dataset without generating intermediate datasets,which saves significant computer memory for large datasets and savescomputer memory, computing time, and communication time for distributeddatasets. Additionally, the user can specify any number oftransformation flows with one or more phases that can be executed inparallel saving significant user time, computer memory, computing time,and communication time. For example, it is common to apply imputation tohandle missing values followed by discretization/binning to handleoutlier values. The workflow can be performed using a singletransformation flow to avoid the generation of intermediate datasets andreduce the number of data passes because the data passes are sharedacross the transformation flows.

It is further beneficial to explore many feature transformation flows.Data transformation application 224, controller data transformationapplication 324, and worker data transformation application 424 easilyand automatically allow the user to evaluate the effect of manytransformation flows in a single execution so that the input dataset ismore effectively evaluated and transformed. For example, variancereducing functional transformations such as Box-Cox anddiscretization/binning can be applied to highly skewed variables in asingle execution in parallel and in the same data pass.

Referring to FIG. 17, a third embodiment of user device 200 is shownthat further includes a high-C (high-cardinality) data transformationapplication 230. High-C data transformation application 230 performsoperations associated with requesting transformation of high-cardinalityvariables identified in the input dataset so that the user can betterutilize the data in subsequent predictive analytics. The operations maybe implemented using hardware, firmware, software, or any combination ofthese methods. Referring to the example embodiment of FIG. 17, high-Cdata transformation application 230 is implemented in software(comprised of computer-readable and/or computer-executable instructions)stored in computer-readable medium 208 and accessible by processor 210for execution of the instructions that embody the operations of high-Cdata transformation application 230. High-C data transformationapplication 230 may be written using one or more programming languages,assembly languages, scripting languages, etc. High-C data transformationapplication 230 may be implemented as a Web application.

High-C data transformation application 230 may be integrated with otheranalytic tools including data analysis application 222 and/or datatransformation application 224. As an example, high-C datatransformation application 230 may be part of an integrated dataanalytics software application and/or software architecture such as thatoffered by SAS Institute Inc. of Cary, N.C., USA. For example, high-Cdata transformation application 230 may be part of SAS® EnterpriseMiner™ developed and provided by SAS Institute Inc. of Cary, N.C., USA.Merely for further illustration, high-C data transformation application230 may be implemented using or integrated with one or more SAS softwaretools such as Base SAS, SAS/STAT®, SAS® High Performance AnalyticsServer, SAS® LASR™, SAS® In-Database Products, SAS® Scalable PerformanceData Engine, SAS/OR®, SAS/ETS®, SAS® Inventory Optimization, SAS®Inventory Optimization Workbench, SAS® Visual Data Mining and MachineLearning, SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statisticsfor Hadoop®, SAS® Forecast Server, all of which are developed andprovided by SAS Institute Inc. of Cary, N.C., USA.

Referring to FIG. 18, a third embodiment of controller device 104 isshown that further includes a controller high-C data transformationapplication 330. Controller high-C data transformation application 330performs operations associated with transforming the input dataset basedon per-level statistics values 332 provided from user device 200 usingthe computing devices of worker system 106, when the input dataset isdistributed across the computing devices of worker system 106. Theoperations may be implemented using hardware, firmware, software, or anycombination of these methods. Referring to the example embodiment ofFIG. 18, controller high-C data transformation application 330 isimplemented in software (comprised of computer-readable and/orcomputer-executable instructions) stored in second computer-readablemedium 308 and accessible by second processor 310 for execution of theinstructions that embody the operations of controller high-C datatransformation application 330. Controller high-C data transformationapplication 330 may be written using one or more programming languages,assembly languages, scripting languages, etc. Controller datatransformation application 330 may be implemented as a Web application.

Controller high-C data transformation application 330 may be integratedwith other analytic tools including with controller data analysisapplication 312 and/or controller data transformation application 324.As an example, controller high-C data transformation application 330 maybe part of an integrated data analytics software application and/orsoftware architecture such as that offered by SAS Institute Inc. ofCary, N.C., USA. For example, controller high-C data transformationapplication 330 may be part of SAS® Enterprise Miner™ developed andprovided by SAS Institute Inc. of Cary, N.C., USA. Merely for furtherillustration, controller high-C data transformation application 330 maybe implemented using or integrated with one or more SAS software toolssuch as Base SAS, SAS/STAT®, SAS® High Performance Analytics Server,SAS® LASR™, SAS® In-Database Products, SAS® Scalable Performance DataEngine, SAS/OR®, SAS/ETS®, SAS® Inventory Optimization, SAS® InventoryOptimization Workbench, SAS® Visual Data Mining and Machine Learning,SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®,SAS® Forecast Server, all of which are developed and provided by SASInstitute Inc. of Cary, N.C., USA.

Referring to FIG. 19, a third embodiment of worker device 400 is shownthat further includes a worker high-C data transformation application430. Worker high-C data transformation application 430 performs datatransformation of high-cardinality variables identified in input datasubset 414 based on inputs from controller device 104 to define subsetper-level statistics values 432 and transformed data subset 428. Workerhigh-C data transformation application 430 may be integrated with otheranalytic tools including worker data analysis application 412 and/orworker data transformation application 424. As an example, worker high-Cdata transformation application 430 may be part of an integrated dataanalytics software application and/or software architecture such as thatoffered by SAS Institute Inc. of Cary, N.C., USA. For example, workerhigh-C data transformation application 430 may be part of SAS®Enterprise Miner™ developed and provided by SAS Institute Inc. of Cary,N.C., USA. Merely for further illustration, worker high-C datatransformation application 430 may be implemented using or integratedwith one or more SAS software tools such as Base SAS, SAS/STAT®, SAS®High Performance Analytics Server, SAS® LASR™, SAS® In-DatabaseProducts, SAS® Scalable Performance Data Engine, SAS/OR®, SAS/ETS®, SAS®Inventory Optimization, SAS® Inventory Optimization Workbench, SAS®Visual Data Mining and Machine Learning, SAS® Visual Analytics, SAS®Viya™, SAS In-Memory Statistics for Hadoop®, SAS® Forecast Server, allof which are developed and provided by SAS Institute Inc. of Cary, N.C.,USA.

High-C data transformation application 230, controller high-C datatransformation application 330, and worker high-C data transformationapplication 430 may be the same or different applications that areintegrated in various manners to perform data transformation ofhigh-cardinality variables identified in the input dataset distributedacross worker system 106. High-C data transformation application 230,controller high-C data transformation application 330, and worker high-Cdata transformation application 430 process a transformation requestcomposed of user specifiable transformation flows. The user can specifymultiple, connected transformation phases per high-C transformation flowin a user configurable order. These transformation phases include amapping method, a hash phase, and/or a cluster phase. Any one of thesephases may be included or excluded in a particular transformation flow.

Most predictive modelling workflows discard high cardinality variablesfrom their predictor variables set because naïve treatment of highcardinality variables, such as one-hot encoding, are fraught withproblems due to explosion of the model dimension, which may in turnresult in model overfitting. In addition, most high cardinalityvariables have rare levels (with very few observations), that make theirnaïve treatment ineffective. However, some high cardinality variablesmay be highly informative. As a result, discarding these variables mayresult in a predictive model that achieves less than optimalperformance. In addition, this problem is compounded by the fact thatdesignating a nominal variable as high versus low cardinality isdependent on an arbitrarily set unique count threshold that itself maybe a problem.

High-C data transformation application 230, controller high-C datatransformation application 330, and worker high-C data transformationapplication 430 provide a scalable (due to single pass standardizedmoment computations and hashing phases) and robust solution to theseproblems especially in the context of regression and classificationproblems. A user specifiable power parameter, p, determines a number offeatures, namely, standardized moments of a target variable that arecomputed for each level of a high cardinality predictor variable. Thesemoments capture a density distribution of the target variable for eachlevel of the high cardinality predictor variable, and hence, can be usedas features in subsequent modelling tasks. The feature generator alsomakes similar count based techniques available for classificationproblems. In some cases, introduction of the new p features to the modelspace may be problematic in predictive modelling problems wheredimensionality is an issue. To alleviate this, the feature generatorprovides a k-means clustering based feature compressor that essentiallymaps the p features into a single cluster identifier based feature. Thisdimensionality reduction benefit is in addition to the benefit that thek-means clustering may provide as an effective feature for the modelingalgorithm. Other clustering methods may be used.

In most cases, high cardinality variables are bound to have levels withvery few observations, namely, sparse levels. These levels complicatethe estimation of any level-based statistics, including standardizedmoments and count statistics. High-C data transformation application230, controller high-C data transformation application 330, and workerhigh-C data transformation application 430 provide two features that canbe used to address this. First, a shrinkage estimator can be used tocompute more accurate estimators. The shrinkage estimator shrinks thelevel-based statistics towards a global estimate with the amount ofshrinkage controlled by the number of observations in the level and auser-definable parameter. Second, hash-based level compression can beused that reduces the cardinality to a more manageable size, therebypotentially decreasing a number of sparse levels.

For an interval target variable, high-C data transformation application230, controller high-C data transformation application 330, and workerhigh-C data transformation application 430 generate p features for eachhigh cardinality variable where p is a user definable parameter thatdenotes a maximum power of a standardized moment. The moments arecomputed for each unique value (level) of the high-cardinality variable.By definition, a value of a first standardized moment is zero, and avalue of a second standardized moment is one. High-C data transformationapplication 230, controller high-C data transformation application 330,and worker high-C data transformation application 430 use a mean and astandard deviation in their place, respectively. For a nominal targetvariable, high-C data transformation application 230, controller high-Cdata transformation application 330, and worker high-C datatransformation application 430 optionally generate a frequency perlevel, an event probability, etc.

High-C data transformation application 230, controller high-C datatransformation application 330, and worker high-C data transformationapplication 430 can compute a shrinkage estimator for the standardizedmoments. A user definable shrinkage hyperparameter controls an amount ofshrinkage that is applied to per-level moment estimators, which isimportant because high cardinality variables may contain levels thathave too few observations for reliable estimation of the per-levelmoments. High-C data transformation application 230, controller high-Cdata transformation application 330, and worker high-C datatransformation application 430 also provide flexible missing valuestreatment by providing an option to transform missing values of the highcardinality variable considering missing values as a unique level.High-C data transformation application 230, controller high-C datatransformation application 330, and worker high-C data transformationapplication 430 also provide level compression of the high cardinalityvariables through hashing by providing a hashing based level compressionthat may be used as an on-the-fly preprocessing step, which is importantfor randomly compressing very high cardinality variables. High-C datatransformation application 230, controller high-C data transformationapplication 330, and worker high-C data transformation application 430also provide task parallel k-means clustering for post-processing tocompress the generated p features into a single cluster identifierfeature. In cases where there are multiple high cardinality inputvariables, a distributed task parallel scheme is implemented in whichindependent k-means clustering tasks are assigned to each worker device400 for processing. This feature compression from p features to one isimportant for problems where model dimensionality needs to beconstrained or limited.

Referring to FIG. 20, example operations associated with high-C datatransformation application 230 are described. Additional, fewer, ordifferent operations may be performed depending on the embodiment. Theorder of presentation of the operations of FIG. 20 is not intended to belimiting. A user can interact with one or more user interface windowspresented to the user in a display under control of data transformationapplication 224 independently or through a browser application in anorder selectable by the user. Although some of the operational flows arepresented in sequence, the various operations may be performed invarious repetitions, concurrently, and/or in other orders than thosethat are illustrated.

A session may be established with controller device 104.

Similar to operation 500, in an operation 2000, the first indicator maybe received that indicates the input dataset.

Similar to operation 502, in an operation 2002, a load of the inputdataset may be requested.

In an operation 2004, a sixth indicator may be received that indicatesone or more high-C transformation flow parameter values. The one or morehigh-C transformation flow parameter values are used to define how eachof one or more high-C variables v_(HCi) are transformed though eachvariable can be transformed multiple times in different high-Ctransformation flows. Each high-C transformation flow parameter value ofthe plurality of high-C transformation flow parameter values may have apredefined default value that may be used when a user does not specify avalue for the high-C transformation flow parameter using the sixthindicator.

In an operation 2006, a request to transform the selected high-Cvariable(s) of the input dataset based on the plurality of high-Ctransformation flow parameter values is sent to controller device 104.For example, the user may select a button to indicate that the pluralityof high-C transformation flow parameter values has been selected andthat transformation of the input dataset should be performed. Theplurality of high-C transformation flow parameter values may be sent ina message or other instruction to controller device 104, may be providedin a known memory location to controller device 104, etc. In analternative embodiment, user device 200 and controller device 104 may beintegrated in the same computing device so that when the plurality ofhigh-C transformation flow parameter values is received by user device200, it is also received by controller device 104.

For example, the following code establishes a session (“session mysess”)and sends a high-C transformation request (“dataPreprocess.transform”)to process the input dataset defined by table={name=“kdd98”where=“target_d>0;”}, which selects a portion of the dataset names“kdd98”.

proc cas; session mysess; dataPreprocess.transform / table ={name=″kdd98″ where=”target_d > 0;”} requestPackages = { { name = ″t1″inputs = ${osource} targets = ${target_d} mapInterval = { method =″moments″ args = { nMoments = 4 includeMissingLevel = True shrinkageFactor = 10 } } } } idVars = ${osource} casout = {name =″out1″ replace=True} ; run; quit;

The transformed dataset is stored in name=“out1”. The transformed valuesreplace the existing high-C variable values though the optionreplace=False adds the transformed values to the existing variablevalues instead of replacing them. In the illustrative code above, the“run” statement triggers sending of the request to controller device104.

The “requestPackages” parameter defines a list of high-C transformationflows that are the plurality of transformation flow parameter valuesthat define at least one high-C transformation flow. Thus, each requestpackage of the “requestPackages” parameter is a high-C transformationflow definition of one or more high-C transformation flow definitions.Because each transformation flow definition can have multiple phases,computation of the parameters for a phase is based on the data thatflows from the preceding phase, if any. In the illustrative code, asingle transformation flow named “t1” is defined by the“requestPackages” parameter. The user can define any number of high-Ctransformation flow definitions. Each high-C transformation flowincludes a “name” parameter that defines a name for the transformationflow and an “inputs” parameter that defines a list of the one or morehigh-C input variables to be transformed by the transformation flow. Forexample, for the first transformation flow definition named “t1”, asingle high-C input variable “osource” is transformed.

A high-C transformation flow definition also includes a “targets”parameter that defines the target variable associated with each high-Cinput variable. For example, for the first transformation flowdefinition named “t1”, the target variable named “target_d” is read fromthe input dataset in association with the single high-C input variable“osource”.

A high-C transformation flow definition also includes a “mapInterval”parameter that defines a transformation method and parameter values. Asan example, a transformation method may be selected from “Moments”,“WOE”, “Frequency Count”, etc. For example, a default transformationmethod may be the Moments transformation method. Of course, thetransformation method may be labeled or selected in a variety ofdifferent manners by the user as understood by a person of skill in theart. In an alternative embodiment, the transformation method may not beselectable, and a single transformation method is implemented by high-Cdata transformation application 230. For example, the Momentstransformation method may be used by default or without allowing aselection. As another example, the transformation method may not bespecified, but may be selected by default based on a data type of thetarget variable specified in operation 2006. For example, the Momentstransformation method may be used by default for a target variable withan interval data type, the WOE transformation method may be used bydefault for a target variable with a binary data type, the FrequencyCount transformation method may be used by default for a target variablewith a nominal (multi-class) data type, etc.

A high-C transformation flow definition also includes a value of anumber of the p features to generate, which may also be referred to as anumber of moments to generate. This is applicable for interval targets.A default value may be stored, for example, in computer-readable medium208 and used automatically. In another alternative embodiment, the valueof the number of the p features to generate may not be selectable.Instead, a fixed, predefined value may be used. For illustration, adefault value may be four.

A high-C transformation flow definition also includes a value of ashrinkage factor. A default value may be stored, for example, incomputer-readable medium 208 and used automatically. In anotheralternative embodiment, the value of the shrinkage factor may not beselectable. Instead, a fixed, predefined value may be used. Forillustration, a default value may be zero.

A high-C transformation flow definition also includes a missing leveloption. The missing level option indicates whether a level is definedwhen a value is missing for the input variable(s). A default value maybe stored, for example, in computer-readable medium 208 and usedautomatically. In another alternative embodiment, the value of themissing level option may not be selectable. Instead, a missing level isalways defined for missing values or is never defined. For illustration,a default value may be “False” to indicate that a missing level is notdefined, and the variable value is skipped.

A high-C transformation flow definition also includes a hash option thatdefines whether hash based level compression is applied as apre-processing step. A default value may be stored, for example, incomputer-readable medium 208 and used automatically. In anotheralternative embodiment, the value of the hash option may not beselectable. Instead, hash based level compression is always applied oris never applied. For illustration, a default value may be “False” toindicate that hash based level compression is not applied.

A high-C transformation flow definition may also include a value of anumber of clusters into which to cluster the p features unless theclustering algorithm determines a number of clusters automatically. Adefault value may be stored, for example, in computer-readable medium208 and used automatically. In another alternative embodiment, the valueof the number of clusters may not be selectable. Instead, a fixed,predefined value may be used. For illustration, a default value may beone. The value of the number of clusters equal to one indicates thatclustering is not performed.

For example, for the first transformation flow named “t1”, thetransformation method selected is the “Moments” transformation method(method=“moments”). The “ergs” parameter indicates that the number ofthe p features to generate is four (nMoments=4), the value of themissing level option is “True” (includeMissingLevel=True), and the valueof the shrinkage factor is ten (shrinkageFactor=10) for the selectedtransformation method. The default value for the value of the number ofclusters into which to cluster the p features is set to the defaultvalue, which in the illustrative embodiment is one so that clustering isnot performed. A high-C transformation flow may be anothertransformation flow read and processed by data transformationapplication 224 when the applications are integrated.

In an operation 2008, a status indicator may be received that indicatesa success or failure of the transformation request. Additionally, or inthe alternative, a summary table may be received that lists transformedvariables.

In an operation 2010, the received status indicator may be presented ondisplay 216.

Referring to FIGS. 21A and 21B, example operations associated withcontroller high-C data transformation application 330 are described.Additional, fewer, or different operations may be performed depending onthe embodiment. The order of presentation of the operations of FIGS. 21Aand 21B is not intended to be limiting. Again, controller high-C datatransformation application 330 and high-C data transformationapplication 230 may be integrated or be the same applications so thatthe operations of FIG. 20 and FIGS. 21A and 21B are merged.Additionally, or in the alternative, controller data analysisapplication 312, controller data transformation application 324, and/orcontroller high-C data transformation application 330 may be integratedor be the same applications so that the operations of FIG. 6, FIGS. 15Aand 15B, and/or FIGS. 21A and 21B are merged.

Similar to operation 600, in an operation 2100, the request to load theinput dataset selected by the user is received.

Similar to operation 602, in an operation 2102, the input dataset isloaded and distributed across each worker device 400 of worker system106.

In an operation 2104, the high-C transformation request may be receivedfrom user device 200 or directly from the user of user device 200 whenintegrated.

In an operation 2106, the plurality of high-C transformation flowparameter values is extracted from the high-C transformation request. Inan alternative embodiment, the request may include a reference to alocation that is storing the values. In another alternative embodiment,the plurality of high-C transformation flow parameter values may be readfrom a known storage location. The plurality of high-C transformationflow parameter values may be stored in transformation flow parametervalues 326.

In an operation 2108, a request is sent to each worker device 400 tocompute per-level statistics for each selected high-C input variable foreach high-C transformation flow. For example, the request includes aninput variable name, a target variable name, and a transformation flowindex for each high-C transformation flow.

In an operation 2110, the per-level statistics results for each selectedhigh-C input variable for each high-C transformation flow are receivedfrom each worker device 400.

In an operation 2112, controller per-level statistics are initializedusing the per-level statistics results from a first worker device 400 ofworker system 106.

In an operation 2114, the per-level statistics results are selected fora next worker device 400 of worker system 106.

In an operation 2116, a first level is selected from the selectedper-level statistics results for the next worker device 400.

In an operation 2118, a determination is made concerning whether thefirst level is included in the initialized controller per-levelstatistics. When the first level is included, processing continues in anoperation 2120. When the first level is not included, processingcontinues in an operation 2122.

In operation 2120, the controller per-level statistics for the firstlevel are updated to include the selected per-level statistics resultsfor the next worker device 400, and processing continues in an operation2124.

In operation 2122, the selected per-level statistics results for thefirst level for the next worker device 400 are copied to the controllerper-level statistics to create the first level in the controllerper-level statistics, and processing continues in operation 2124.

In operation 2124, a determination is made concerning whether theselected per-level statistics results for the next worker device 400include another level. When there is another level, processing continuesin operation 2126. When there is not another level, processing continuesin an operation 2128.

In operation 2126, the next level is selected from the selectedper-level statistics results for the next worker device 400, andprocessing continues in operation 2118.

In operation 2128, a determination is made concerning whether there isanother worker device 400 of worker system 106 to process. When there isanother worker device 400, processing continues in operation 2130. Whenthere is not another worker device 400, processing continues in anoperation 2132.

In operation 2130, a next worker device 400 is selected from workersystem 106, and processing continues in operation 2114.

Referring to FIG. 21B, in an operation 2134, a determination is madeconcerning whether the value of the shrinkage factor is greater thanzero. When the value of the shrinkage factor is greater than zero,processing continues in operation 2136. When the value of the shrinkagefactor is not greater than zero, processing continues in an operation2146.

In an operation 2136, global standardized moments are computed from thecontroller per-level statistics using the method described in PhillippePébay, Formulas for Robust, One-Pass Parallel Computation of Covariancesand Arbitrary-Order Statistical Moments, Sandia Report SAND2008-6212,Sandia National Laboratories (2008).

In an operation 2138, a first level is selected from the controllerper-level statistics.

In an operation 2140, the value of the shrinkage factor is applied tothe first level of the controller per-level statistics using the methoddescribed in J. B. Copas, Regression, Prediction and Shrinkage (withDiscussion), 45 Journal of the Royal Statistical Society SeriesB-Methodological 311 (1983). The estimated standardized moments for thelevels with observations fewer than the value of the shrinkage factor ismade very close to the global standardized moments using the method.

In operation 2142, a determination is made concerning whether thecontroller per-level statistics include another level. When there isanother level, processing continues in operation 2144. When there is notanother level, processing continues in an operation 2146.

In operation 2144, the next level is selected from the controllerper-level statistics, and processing continues in operation 2140.

In operation 2146, a determination is made concerning whether clusteringwas selected. For example, when the value of the number of clusters intowhich to cluster the p features is greater than one, clustering wasselected. When clustering was selected, processing continues in anoperation 2148. When clustering was not selected, processing continuesin an operation 2150.

In operation 2148, a number of k-means clustering tasks is initializedto the number of high cardinality input variables selected, and asequential task identifier is assigned to each k-means clustering task.

In an operation 2150, the clustering tasks are assigned to worker device400 of worker system 106 in a round-robin fashion until each clusteringtask has been assigned. A list of task identifier and worker deviceidentifier of the assigned worker device 400 may be created.

In an operation 2152, clustering of the assigned task (variable) isrequested of each worker device 400 of worker system 106. The requestsent to each worker device 400 may include the controller per-levelstatistics for the assigned variable that is to be clustered.

In an operation 2154, a cluster assignment for each level of thecontroller per-level statistics for the assigned variable is receivedfrom each worker device 400 of worker system 106. The cluster assignmentassigns a cluster identifier to each level of the controller per-levelstatistics for the assigned variable.

In an operation 2156, a request is sent to each worker device 400 totransform each input variable for each high-C transformation flow usingthe cluster identifier assigned to each level for each high-C variable.

In an operation 2158, a request is sent to each worker device 400 totransform each input variable for each high-C transformation flow usingthe controller per-level statistics that may have been shrunk and/orcompressed.

In an operation 2160, a done indicator is received from each workerdevice 400.

In operation 2162, a done indicator is sent to user device 200. Thetransformed data may be stored in transformed data subset 428 at eachworker device 400.

Referring to FIGS. 22A and 22B, example operations associated withworker high-C data transformation application 430 are described.Additional, fewer, or different operations may be performed depending onthe embodiment. The order of presentation of the operations of FIGS. 22Aand 22B not intended to be limiting. Controller high-C datatransformation application 330 and worker high-C data transformationapplication 430 may be integrated or be the same applications so thatthe operations of FIGS. 21A and 21B and FIGS. 22A and 22B are merged.

Additionally, or in the alternative, worker data analysis application412, worker data transformation application 424 and/or worker high-Cdata transformation application 430 may be integrated or be the sameapplications so that the operations of FIGS. 7A and 7B, FIGS. 16A, 16B,and 16C, and/or FIGS. 22A and 22B are merged.

Similar to operation 700, in an operation 2200, a portion of the inputdataset is received and loaded in third computer-readable medium 408 asinput data subset 414.

In an operation 2202, the request to compute per-level statistics valuesis received. The request may include an indicator of the high-Ctransformation flow, the high-C input variable, the target variableassociated with the high-C input variable, the number of the p featuresto generate for the high-C input variable, and/or the value of themissing level option for each high-C transformation flow.

In an operation 2204, an observation is read from input data subset 414to define values for each high-C input variable of each high-Ctransformation flow and the target variable associated with each high-Cinput variable. Hash based level compression is applied if selected bythe user as indicated by the plurality of high-C transformation flowparameter values. When hash based level compression is applied, for eachvalue of a variable, a hash function is applied to map the nominal valueto an integer index by taking the remainder (modulo operator) of thehashed value with the number of buckets of the hash table to limit anumber of distinct levels to the number of buckets, which may beuser-definable.

In an operation 2206, a current input value of a first high-C inputvariable and a target value of the target variable associated with thefirst high-C input variable are selected from the read observation.

In operation 2208, a determination is made concerning whether thecurrent input value is a new level for the high-C input variable. Whenthe value is a new level, processing continues in an operation 2210.When the value is not a new level, processing continues in an operation2212.

In operation 2210, statistical data values are initialized for each ofthe p features to generate. For example, counters and sum values areinitialized to zero. Minimum values may be initialized to a largequantity, and maximum values may be initialized to a large negativequantity. Illustrative counters include a number of observations countervalue, a number of missing observations counter value, a number ofunique values counter value, a number of occurrences of each uniquevalue counter value, etc. Illustrative sum values include a total sum ofvalues of each variable, a total sum of squared values of each variable,etc.

In an operation 2212, the statistical data values for each of the pfeatures to generate for the level are selected.

In an operation 2214, the statistical data values for each of the pfeatures to generate for the level are updated using

$M_{p,\zeta} = {M_{p,\zeta_{1}} + {\sum\limits_{k = 1}^{p - 2}\; {\begin{pmatrix}k \\p\end{pmatrix}{M_{{p - k},\zeta_{1}}( \frac{- \delta}{n} )}^{k}}} + {( \frac{( {n - 1} )\delta}{n} )^{p}\lbrack {1 - ( \frac{- 1}{n - 1} )^{p - 1}} \rbrack}}$

where δ=y−μ₁, where y is the target value, μ₁ is a mean value, n is anumber of observations, M is the statistical data value for the featureor moment that is one of the p features to generate, ζ₁ indicates thestatistical data value without a contribution from the new observation,and ζ indicates the statistical data value with the contribution fromthe new observation.

In operation 2216, a determination is made concerning whether there isanother high-C input variable. When there is another high-C inputvariable, processing continues in an operation 2218. When there is notanother high-C input variable, processing continues in an operation2220.

In operation 2218, a current input value of a next high-C input variableand a target value of the target variable associated with the nexthigh-C input variable are selected from the read observation, andprocessing continues in operation 2208.

In operation 2220, a determination is made concerning whether there isanother observation in input data subset 414. When there is anotherobservation, processing continues in an operation 2204. When there isnot another observation, processing continues in an operation 2222.

In operation 2222, the updated level statistical data value(s) for eachof the p features to generate are returned or otherwise provided tocontroller device 104. The updated level statistical data value(s) maybe stored in subset per-level statistics values 432.

Referring to FIG. 22B, in an operation 2224, a clustering request isreceived that includes the controller per-level statistics value foreach of the p features to generate for an assigned variable. Eachclustering task is defined by a contingency table that contains thelevel and per-level statistics of the variable that defines the task.For illustration, Table V below depicts a slice of input to the k-meansclustering task for an assigned variable where p=4:

TABLE V Moment 1 Moment 2 Moment 3 Moment 4 level (mean) (std. dev.)(third moment) (fourth moment) 1 0.5 1 12 14 2 0.6 2.5 3 120 3 10 120131 1400 . . . . . . . . . . . . . . .

In an operation 2226, k-means clustering (or another type of clustering)is performed to map each level to a cluster identifier. Forillustration, the paper by Hartigan, J. A. and Wong, M. A., Algorithm AS136: A K-Means Clustering Algorithm (1979) describes a k-meansclustering method.

In an operation 2228, the per-level cluster assignments are returned tocontroller device 104.

In an operation 2230, the request to transform each high-C inputvariable of each high-C transformation flow is received. The request mayinclude an indicator of the high-C transformation flow, the inputvariable, and the per-level cluster identifier for each high-C inputvariable and for each high-C transformation flow when the request isreceived as a result of execution of operation 2156. The request mayinclude an indicator of the high-C transformation flow, the inputvariable, and the controller per-level statistics value for each of thep features for each high-C input variable and for each high-Ctransformation flow when the request is received as a result ofexecution of operation 2158.

Transformed data subset 428 may be opened for writing on a first row. Aheader row may be written to the first row of transformed data subset428 that includes a variable name for each transformed variable. Forexample, the variable name for each variable to transform may beappended to the “name” parameter value given to each high-Ctransformation flow so that each transformed variable has a unique name.For example, in the illustrative code above, a first transformedvariable may be named “t1_osource_1” for a first feature of the pfeatures or the cluster identifier, a second transformed variable may benamed “t1_osource_2” for a second feature of the p features, a thirdtransformed variable may be named “t1_osource_3” for a third feature ofthe p features, etc.

In an operation 2232, an observation is read as a current observationfrom input data subset 414 to define values for each high-C variable ofeach high-C transformation flow. When the option replace=False isselected by the user, the current observation may be written totransformed data subset 428 so that transformed values are appended tothe original values read from input data subset 414.

In an operation 2234 a current input value of a first high-C variable isselected as a current value from the read observation.

In an operation 2236, either the per-level statistics or the clusteridentifier are selected based on the current input value from the valuesreceived in the request.

In an operation 2238, either the selected per-level statistics or theselected cluster identifier are appended to the current row oftransformed data subset 428 as an output value of the current high-Ctransformation flow for the value of the current variable.

In operation 2240, a determination is made concerning whether there isanother high-C input variable to transform. When there is another high-Cinput variable, processing continues in an operation 2242. When there isnot another high-C input variable, processing continues in an operation2244.

In operation 2242, a current input value of a next high-C input variableis selected from the read observation, and processing continues inoperation 2236.

In operation 2244, a determination is made concerning whether there isanother observation in input data subset 414. When there is anotherobservation, processing continues in operation 2232. When there is notanother observation, processing continues in an operation 2246.

In operation 2246, a done indicator is sent to controller device 104.

Referring to FIG. 23, a fourth embodiment of user device 200 is shownthat further includes a training application 240. Training application240 performs operations associated with training a model usingtransformed data subset 428. The operations may be implemented usinghardware, firmware, software, or any combination of these methods.Referring to the example embodiment of FIG. 23, training application 240is implemented in software (comprised of computer-readable and/orcomputer-executable instructions) stored in computer-readable medium 208and accessible by processor 210 for execution of the instructions thatembody the operations of training application 240. Training application240 may be written using one or more programming languages, assemblylanguages, scripting languages, etc. Training application 240 may beimplemented as a Web application.

Training application 240 may be integrated with other analytic toolsincluding data analysis application 222, data transformation application224, and/or high-C data transformation application 230. As an example,training application 240 may be part of an integrated data analyticssoftware application and/or software architecture such as that offeredby SAS Institute Inc. of Cary, N.C., USA. For example, trainingapplication 240 may be part of SAS® Enterprise Miner™ developed andprovided by SAS Institute Inc. of Cary, N.C., USA. Merely for furtherillustration, training application 240 may be implemented using orintegrated with one or more SAS software tools such as Base SAS,SAS/STAT®, SAS® High Performance Analytics Server, SAS® LASR™, SAS®In-Database Products, SAS® Scalable Performance Data Engine, SAS/OR®,SAS/ETS®, SAS® Inventory Optimization, SAS® Inventory OptimizationWorkbench, SAS® Visual Data Mining and Machine Learning, SAS® VisualAnalytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®, SAS®Forecast Server, all of which are developed and provided by SASInstitute Inc. of Cary, N.C., USA.

Training application 240 performs operations associated with definingpredictive model parameters 242 from data stored in transformed datasubset 428 distributed across worker system 106. Predictive modelparameters 242 may be used to classify data stored in a scoring dataset2524 (shown referring to FIG. 25) to support various data analysisfunctions as well as provide alert/messaging related to the classifieddata. Some or all of the operations described herein may be embodied intraining application 240. The operations may be implemented usinghardware, firmware, software, or any combination of these methods.

Training application 240 may be integrated with other system processingtools to automatically process data generated as part of operation of anenterprise, device, system, facility, etc., to identify any outliers inthe processed data, to monitor changes in the data, and to provide awarning or alert associated with the monitored data using inputinterface 202, output interface 204, and/or communication interface 206so that appropriate action can be initiated in response to changes inthe monitored data.

Referring to FIG. 24, example operations associated with trainingapplication 240 are described. Additional, fewer, or differentoperations may be performed depending on the embodiment. The order ofpresentation of the operations of FIG. 24 is not intended to belimiting. A user can interact with one or more user interface windowspresented to the user in a display under control of data transformationapplication 240 independently or through a browser application in anorder selectable by the user. Although some of the operational flows arepresented in sequence, the various operations may be performed invarious repetitions, concurrently, and/or in other orders than thosethat are illustrated.

A session may be established with controller device 104.

Similar to operation 500, in an operation 2400, the first indicator maybe received that indicates the input dataset. The input dataset may bean indicator of the transformed dataset.

Similar to operation 502, in an operation 2402, a load of the inputdataset may be requested.

In an operation 2404, a seventh indicator of a model definition isreceived. For example, the model definition indicates a model type totrain and any hyperparameters to use as part of the model training.Illustrative model types include a neural network model type, a gradientboosting tree model type, a decision tree model type, a forest modeltype, a support vector machine model type, etc.

In an operation 2406, a request to train a model of the specified modeltype with the input dataset is sent to controller device 104.

In an operation 2408, results of training the model type are receivedand stored in predictive model parameters 242. The results describe apredictive model. The results may be an analytic store created using theASTORE procedure provided by the SAS Visual Data Mining and MachineLearning Procedures developed and provided by SAS Institute Inc. ofCary, N.C., USA.

Referring to FIG. 25, a block diagram of a prediction device 2500 isshown in accordance with an illustrative embodiment. Prediction device2500 may include a fourth input interface 2502, a fourth outputinterface 2504, a fourth communication interface 2506, a fourthnon-transitory computer-readable medium 2508, a fourth processor 2510, aprediction application 2522, predictive model parameters 242, scoringdataset 2524, and predicted dataset 2526. Fewer, different, and/oradditional components may be incorporated into prediction device 2500.Prediction device 2500 and user device 200 and/or controller device 104may be the same or different devices.

Fourth input interface 2502 provides the same or similar functionalityas that described with reference to input interface 202 of user device200 though referring to prediction device 2500. Fourth output interface2504 provides the same or similar functionality as that described withreference to output interface 204 of user device 200 though referring toprediction device 2500. Fourth communication interface 2506 provides thesame or similar functionality as that described with reference tocommunication interface 206 of user device 200 though referring toprediction device 2500. Data and messages may be transferred betweenprediction device 2500 and a distributed computing system 2528 usingfourth communication interface 2506. Fourth computer-readable medium2508 provides the same or similar functionality as that described withreference to computer-readable medium 208 of user device 200 thoughreferring to prediction device 2500. Fourth processor 2510 provides thesame or similar functionality as that described with reference toprocessor 210 of user device 200 though referring to prediction device2500.

Prediction application 2522 performs operations associated withclassifying or predicting a characteristic of each observation ofscoring dataset 2524 that is stored in predicted dataset 2526 to supportvarious data analysis functions as well as provide alert/messagingrelated to the classified/predicted data. Dependent on the type of datastored in the input dataset and scoring dataset 2524, predictionapplication 2522 may identify anomalies as part of process control, forexample, of a manufacturing process, for machine condition monitoring,for example, an electro-cardiogram device, for image classification, forintrusion detection, for fraud detection, etc. Some or all of theoperations described herein may be embodied in prediction application2522. The operations may be implemented using hardware, firmware,software, or any combination of these methods.

Referring to the example embodiment of FIG. 25, prediction application2522 is implemented in software (comprised of computer-readable and/orcomputer-executable instructions) stored in fourth computer-readablemedium 2508 and accessible by fourth processor 2510 for execution of theinstructions that embody the operations of prediction application 2522.Prediction application 2522 may be written using one or more programminglanguages, assembly languages, scripting languages, etc. Predictionapplication 2522 may be integrated with other analytic tools. As anexample, prediction application 2522 may be part of an integrated dataanalytics software application and/or software architecture such as thatoffered by SAS Institute Inc. of Cary, N.C., USA. For example,prediction application 2522 may be part of SAS® Enterprise Miner™developed and provided by SAS Institute Inc. of Cary, N.C., USA. Merelyfor further illustration, prediction application 2522 may be implementedusing or integrated with one or more SAS software tools such as BaseSAS, SAS/STAT®, SAS® High Performance Analytics Server, SAS® LASR™, SAS®In-Database Products, SAS® Scalable Performance Data Engine, SAS/OR®,SAS/ETS®, SAS® Inventory Optimization, SAS® Inventory OptimizationWorkbench, SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statisticsfor Hadoop®, SAS® Forecast Server, all of which are developed andprovided by SAS Institute Inc. of Cary, N.C., USA. One or moreoperations of prediction application 2522 further may be performed by anESPE. Prediction application 2522, data analysis application 222,controller data analysis application 312, worker data analysisapplication 412, and/or training application 240 may be the same ordifferent applications that are integrated in various manners totransform data used to train and execute a model using scoring dataset2524.

Prediction application 2522 may be implemented as a Web application.Prediction application 2522 may be integrated with other systemprocessing tools to automatically process data generated as part ofoperation of an enterprise, to classify data in the processed data,and/or to provide a warning or alert associated with the dataclassification using fourth input interface 2502, fourth outputinterface 2504, and/or fourth communication interface 2506 so thatappropriate action can be initiated in response. For example, a warningor an alert may be presented using a second display 2516, a secondspeaker 2518, a second printer 2520, etc. or sent to one or morecomputer-readable media, display, speaker, printer, etc. of distributedcomputing system 2528.

The input dataset and scoring dataset 2524 may be generated, stored, andaccessed using the same or different mechanisms. Similar to the inputdataset, scoring dataset 2524 may include a plurality of rows and aplurality of columns with the plurality of rows referred to asobservations or records, and the columns referred to as variables thatare associated with an observation. Scoring dataset 2524 may betransposed.

Scoring dataset 2524 may be stored on fourth computer-readable medium2508 or on one or more computer-readable media of distributed computingsystem 2528 and accessed by prediction device 2500 using fourthcommunication interface 2506. Data stored in scoring dataset 2524 may bea sensor measurement or a data communication value, for example, from asensor 2513, may be generated or captured in response to occurrence ofan event or a transaction, generated by a device such as in response toan interaction by a user with the device, for example, from a secondkeyboard 2512 or a second mouse 2514, etc. The data stored in scoringdataset 2524 may include any type of content represented in anycomputer-readable format such as binary, alphanumeric, numeric, string,markup language, etc. The content may include textual information,graphical information, image information, audio information, numericinformation, etc. that further may be encoded using various encodingtechniques as understood by a person of skill in the art. The datastored in scoring dataset 2524 may be captured at different time pointsperiodically, intermittently, when an event occurs, etc. One or morecolumns may include a time value. Similar to the input dataset, datastored in scoring dataset 2524 may be generated as part of the IoT, andsome or all data may be pre- or post-processed by an ESPE.

Scoring dataset 2524 may be stored in various compressed formats such asa coordinate format, a compressed sparse column format, a compressedsparse row format, etc. Scoring dataset 2524 further may be stored usingvarious structures as known to those skilled in the art including a filesystem, a relational database, a system of tables, a structured querylanguage database, etc. on prediction device 2500 and/or on predictionapplication 2522. Prediction device 2500 and/or prediction application2522 may coordinate access to scoring dataset 2524 that is distributedacross worker system 106 and/or controller device 104. For example,scoring dataset 2524 may be stored in a cube distributed across a gridof computers as understood by a person of skill in the art. As anotherexample, scoring dataset 2524 may be stored in a multi-node Hadoop®cluster. As another example, scoring dataset 2524 may be stored in acloud of computers and accessed using cloud computing technologies, asunderstood by a person of skill in the art. The SAS® LASR™ AnalyticServer and/or SAS® Viya™ may be used as an analytic platform to enablemultiple users to concurrently access data stored in scoring dataset2524.

Referring to FIG. 26, example operations of prediction application 2522are described. Additional, fewer, or different operations may beperformed depending on the embodiment of prediction application 2522.The order of presentation of the operations of FIG. 26 is not intendedto be limiting. Although some of the operational flows are presented insequence, the various operations may be performed in variousrepetitions, concurrently (in parallel, for example, using threadsand/or a distributed computing system), and/or in other orders thanthose that are illustrated.

In an operation 2600, an eighth indicator may be received that indicatesscoring dataset 2524. For example, the eighth indicator indicates alocation and a name of scoring dataset 2524. As an example, the eighthindicator may be received by prediction application 2522 after selectionfrom a user interface window or after entry by a user into a userinterface window. In an alternative embodiment, scoring dataset 2524 maynot be selectable. For example, a most recently created dataset may beused automatically.

In an operation 2602, a ninth indicator may be received that indicatespredictive model parameters 242. For example, the ninth indicatorindicates a location and a name of predictive model parameters 242. Asan example, the ninth indicator may be received by predictionapplication 2522 after selection from a user interface window or afterentry by a user into a user interface window. In an alternativeembodiment, predictive model parameters 242 may not be selectable. Forexample, a most recently created model configuration data may be usedautomatically. As another example, predictive model parameters 242 maybe provided automatically as part of integration with trainingapplication 240.

In an operation 2604, a model configuration is read from predictivemodel parameters 242.

In an operation 2606, a model is instantiated with the read modelconfiguration. For example, the type of model, its hyperparameters, andother characterizing elements are read and used to instantiate the modeltrained using the transformed input dataset.

In an operation 2608, an observation vector is read from scoring dataset2524.

In an operation 2610, the observation vector is pre-processed, if anypre-processing is performed.

In an operation 2612, the optionally pre-processed observation vector isinput to the instantiated model.

In an operation 2614, an output of the instantiated model is received.The output may indicate a predicted characteristic computed from theobservation vector using the instantiated model.

In an operation 2616, the predicted characteristic may be output, forexample, by storing the predicted characteristic with the observationvector to predicted dataset 2526. In addition, or in the alternative,the predicted characteristic may be presented on second display 2516,printed on second printer 2520, sent to another computing device usingfourth communication interface 2506, an alarm or other alert signal maybe sounded through second speaker 2518, etc.

In an operation 2618, a determination is made concerning whether scoringdataset 2524 includes another observation vector. When scoring dataset2524 includes another observation vector, processing continues in anoperation 2620. When scoring dataset 2524 does not include anotherobservation vector, processing continues in an operation 2622.

In operation 2620, a next observation vector is read from scoringdataset 2524, and processing continues in operation 2610.

In operation 2622, processing stops and cleanup is performed as needed.

The explosion of digital data is generating many opportunities for bigdata analytics, which in turn provides many opportunities for analyzingthe data and grouping variables to capitalize on the informationcontained in the data—to make better predictions that lead to betterdecisions.

Data analysis and transformation system 100 outputs a hierarchicalvariable grouping in which the groups are characterized by comprehensiveand multi-dimensional statistical metrics that can be consumed indownstream analytics by systems that perform pipelined variabletransformations. An example of such a system is data transformationapplication 224, controller data transformation application 324, andworker data transformation application 424. Additionally, data analysisand transformation system 100 can generate a complete array ofmeta-learning dataset features beneficial to meta-learning systems asthese features can capture salient features of datasets that aredifficult to capture with traditional, individual (non-interacting)features. Data analysis and transformation system 100 also can be usedfor effective visualization of data quality problems in modern datasetsthat are typically characterized by large dimensions, which is importantbecause it helps the user select the proper algorithms for applicationin downstream analytics.

Some machine-learning approaches may be more efficiently and speedilyexecuted and processed with machine-learning specific processors (e.g.,not a generic CPU). Such processors may also provide an energy savingswhen compared to generic CPUs. For example, some of these processors caninclude a graphical processing unit (GPU), an application-specificintegrated circuit (ASIC), a field-programmable gate array (FPGA), anartificial intelligence (AI) accelerator, a neural computing core, aneural computing engine, a neural processing unit, a purpose-built chiparchitecture for deep learning, and/or some other machine-learningspecific processor that implements a machine learning approach or one ormore neural networks using semiconductor (e.g., silicon (Si), galliumarsenide (GaAs)) devices. These processors may also be employed inheterogeneous computing architectures with a number of and a variety ofdifferent types of cores, engines, nodes, and/or layers to achievevarious energy efficiencies, processing speed improvements, datacommunication speed improvements, and/or data efficiency targets andimprovements throughout various parts of the system.

The word “illustrative” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“illustrative” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Further, for the purposes ofthis disclosure and unless otherwise specified, “a” or “an” means “oneor more”. Still further, using “and” or “or” in the detailed descriptionis intended to include “and/or” unless specifically indicated otherwise.The illustrative embodiments may be implemented as a method, apparatus,or article of manufacture using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof to control a computer to implement the disclosed embodiments.

The foregoing description of illustrative embodiments of the disclosedsubject matter has been presented for purposes of illustration and ofdescription. It is not intended to be exhaustive or to limit thedisclosed subject matter to the precise form disclosed, andmodifications and variations are possible in light of the aboveteachings or may be acquired from practice of the disclosed subjectmatter. The embodiments were chosen and described in order to explainthe principles of the disclosed subject matter and as practicalapplications of the disclosed subject matter to enable one skilled inthe art to utilize the disclosed subject matter in various embodimentsand with various modifications as suited to the particular usecontemplated.

1.-30. (canceled)
 31. A non-transitory computer-readable medium havingstored thereon computer-readable instructions that when executed by acomputing device cause the computing device to: receive a transformationflow definition, wherein the transformation flow definition includes oneor more flow variables and a transformation method to apply to the oneor more flow variables, wherein each flow variable of the one or moreflow variables is a high-cardinality variable based on a number ofunique values of a respective flow variable of the one or more flowvariables; request per-level statistics analysis of an input dataset byeach of a plurality of worker computing devices, wherein the inputdataset is distributed across the plurality of worker computing devices;(a) read an observation vector from a subset of the input datasetdistributed to a respective worker computing device to define a firstvariable value for each variable of the one or more flow variables; (b)select a variable of the one or more flow variables as a currentvariable; (c) define a current value equal to the defined first variablevalue for the current variable; (d) determine if the defined currentvalue is a new level or an existing level for the current variable; (e)when the defined current value is the new level, initialize a statisticvalue in association with the new level based on the defined currentvalue and the transformation method to define per-level statistics forthe current variable; (f) when the defined current value is the existinglevel, update the statistic value from the defined per-level statisticsfor the current variable in association with the existing level based onthe defined current value and the transformation method; (g) repeat (c)to (f) with each remaining variable of the one or more flow variables asthe current variable; (h) repeat (a) to (g) with each remainingobservation vector from the subset of the input dataset distributed tothe respective worker computing device, wherein (a) to (h) is performedby each worker computing device of the plurality of worker computingdevices as the respective worker computing device; receive the definedper-level statistics for each variable of the one or more flow variablesfrom each worker computing device of the plurality of worker computingdevices, wherein each level of the defined per-level statistics has anassociated statistic value; define controller per-level statistics fromthe defined per-level statistics for each variable of the one or moreflow variables received from each worker computing device of theplurality of worker computing devices; request transformation of eachvariable of the one or more flow variables by each of the plurality ofworker computing devices using the defined controller per-levelstatistics; (i) repeat the read of the observation vector from thesubset of the input dataset distributed to the respective workercomputing device to define the first variable value for each variable ofthe one or more flow variables, (j) select a second variable of the oneor more flow variables as the current variable; (k) define the currentvalue equal to the defined first variable value for the currentvariable; (l) select the statistic value associated with the definedcurrent value from the respective level of the defined controllerper-level statistics; (m) output the selected statistic value to atransformed input dataset to replace the defined current value; (n)repeat (k) to (m) with each remaining variable of the one or more flowvariables as the current variable; and (o) repeat (i) to (n) with eachremaining observation vector from the subset of the input datasetdistributed to the respective worker computing device, wherein (i) to(o) is performed by each worker computing device of the plurality ofworker computing devices as the respective worker computing device. 32.The non-transitory computer-readable medium of claim 31, wherein aplurality of transformation flow definitions is received.
 33. Thenon-transitory computer-readable medium of claim 32, wherein a firstvariable of a first transformation flow definition of the plurality oftransformation flow definitions and a second variable of a secondtransformation flow definition of the plurality of transformation flowdefinitions are identical.
 34. The non-transitory computer-readablemedium of claim 33, wherein the transformation method to apply to thefirst variable of the first transformation flow definition is differentfrom the transformation method to apply to the second variable of thesecond transformation flow definition.
 35. The non-transitorycomputer-readable medium of claim 32, wherein the computer-readableinstructions further cause the computing device to: (p) select atransformation flow definition from the plurality of transformation flowdefinitions as a current transformation flow after (a) and before (b),wherein the variable is selected from the one or more flow variables ofthe current transformation flow; and (q) repeat (b) to (g) with eachremaining transformation flow definition of the plurality oftransformation flow definitions as the current transformation flow after(g) and before (h), wherein the current variable selected in (b) isselected from the one or more flow variables of the currenttransformation flow and each remaining variable in (g) is selected fromthe one or more flow variables of the current transformation flow,wherein (p) and (q) are further repeated in (h).
 36. The non-transitorycomputer-readable medium of claim 35, wherein the computer-readableinstructions further cause the computing device to: (r) select thetransformation flow definition from the plurality of transformation flowdefinitions as the current transformation flow after (i) and before (j),wherein the variable is selected from the one or more flow variables ofthe current transformation flow; and (s) repeat (j) to (n) with eachremaining transformation flow definition of the plurality oftransformation flow definitions as the current transformation flow after(n) and before (o), wherein the current variable selected in (j) isselected from the one or more flow variables of the currenttransformation flow and each remaining variable in (n) is selected fromthe one or more flow variables of the current transformation flow,wherein (r) and (s) are further repeated in (o).
 37. The non-transitorycomputer-readable medium of claim 31, wherein defining controllerper-level statistics from the defined per-level statistics comprises:selecting a first worker computing device from the plurality of workercomputing devices as a current worker computing device; for each levelof the per-level statistics received from the current worker computingdevice, copying a respective level and the statistic value associatedwith the respective level for each variable of the one or more flowvariables to the controller per-level statistics; selecting a nextworker computing device from the plurality of worker computing devicesas the current worker computing device; (aa) for each level of theper-level statistics and for each variable of the one or more flowvariables received from the current worker computing device, determiningif the respective level of each variable of the one or more flowvariables is a new level of the controller per-level statistics or anexisting level of the controller per-level statistics; when therespective level is the new level of the defined controller per-levelstatistics, copying the respective level and the statistic valueassociated with the respective level to the controller per-levelstatistics for a respective variable of the one or more flow variables;and when the respective level is the existing level of the controllerper-level statistics, updating the statistic value associated with theexisting level of the controller per-level statistics for the respectivevariable of the one or more flow variables based on the statistic valueassociated with the respective level received from the current workercomputing device; and repeating (aa) with each remaining workercomputing device of the plurality of worker computing devices as thecurrent worker computing device.
 38. The non-transitorycomputer-readable medium of claim 31, wherein, after (c) and before (d),the computer-readable instructions further cause the computing device tocompute a hash value using hash based level compression applied to thefirst variable value for each variable of the one or more flow variablesbased on the transformation flow definition to replace the firstvariable value with the computed hash value.
 39. The non-transitorycomputer-readable medium of claim 31, wherein, after (h) and beforerequesting transformation of each variable of the one or more flowvariables by each of the plurality of worker computing devices using thedefined controller per-level statistics, the computer-readableinstructions further cause the computing device to compute a globalstandardized per-level statistic value from the defined controllerper-level statistics and to apply a shrinkage factor to the computedglobal standardized per-level statistic value for each level of thedefined controller per-level statistics.
 40. The non-transitorycomputer-readable medium of claim 31, wherein, after (h) and beforerequesting transformation of each variable of the one or more flowvariables by each of the plurality of worker computing devices using thedefined controller per-level statistics, the computer-readableinstructions further cause the computing device to request clustering ofeach level of the defined controller per-level statistics of eachvariable of the one or more flow variables by the plurality of workercomputing devices to define a cluster identifier for each level.
 41. Thenon-transitory computer-readable medium of claim 31, wherein thestatistic value is updated in (f) using$M_{p,\zeta} = {M_{p,\zeta_{1}} + {\sum\limits_{k = 1}^{p - 2}\; {\begin{pmatrix}k \\p\end{pmatrix}{M_{{p - k},\zeta_{1}}( \frac{- \delta}{n} )}^{k}}} + {( \frac{( {n - 1} )\delta}{n} )^{p}\lbrack {1 - ( \frac{- 1}{n - 1} )^{p - 1}} \rbrack}}$where p is a predefined number of features to generate for the existinglevel, δ=y−μ₁, where y is a target value of the existing level, and μ₁is a mean value, n is a number of observation values included for theexisting level, M is the statistic value, ζ₁ indicates the statisticvalue without a contribution from the defined current value, and ζindicates the statistic value with the contribution from the definedcurrent value.
 42. A computing system comprising: a controller computingdevice comprising a controller processor; and a controllercomputer-readable medium operably coupled to the controller processor,the controller computer-readable medium having controllercomputer-readable instructions stored thereon that, when executed by thecontroller processor, cause the controller computing device to receive atransformation flow definition, wherein the transformation flowdefinition includes one or more flow variables and a transformationmethod to apply to the one or more flow variables, wherein each flowvariable of the one or more flow variables is a high-cardinalityvariable based on a number of unique values of a respective flowvariable of the one or more flow variables; request per-level statisticsanalysis of an input dataset by each of a plurality of worker computingdevices, wherein the input dataset is distributed across the pluralityof worker computing devices; receive defined per-level statistics foreach variable of the one or more flow variables from each workercomputing device of the plurality of worker computing devices, whereineach level of the defined per-level statistics has an associatedstatistic value; define controller per-level statistics from the definedper-level statistics for each variable of the one or more flow variablesreceived from each worker computing device of the plurality of workercomputing devices; request transformation of each variable of the one ormore flow variables by each of the plurality of worker computing devicesusing the controller per-level statistics; and the plurality of workercomputing devices, wherein each worker computing device of the pluralityof worker computing devices comprises a second processor; and a secondnon-transitory computer-readable medium operably coupled to the secondprocessor, the second computer-readable medium having workercomputer-readable instructions stored thereon that, when executed by thesecond processor, cause a respective worker computing device to (a) readan observation vector from a subset of the input dataset distributed tothe respective worker computing device to define a first variable valuefor each variable of the one or more flow variables; (b) select avariable of the one or more flow variables as a current variable; (c)define a current value equal to the defined first variable value for thecurrent variable; (d) determine if the defined current value is a newlevel or an existing level for the current variable; (e) when thedefined current value is the new level, initialize a statistic value inassociation with the new level based on the defined current value andthe transformation method to define per-level statistics for the currentvariable; (f) when the defined current value is the existing level,update the statistic value from the defined per-level statistics for thecurrent variable in association with the existing level based on thedefined current value and the transformation method; (g) repeat (c) to(f) with each remaining variable of the one or more flow variables asthe current variable; (h) repeat (a) to (g) with each remainingobservation vector from the subset of the input dataset distributed tothe respective worker computing device; (i) repeat the read of theobservation vector from the subset of the input dataset distributed tothe respective worker computing device to define the first variablevalue for each variable of the one or more flow variables, (j) select asecond variable of the one or more flow variables as the currentvariable; (k) define the current value equal to the defined firstvariable value for the current variable; (l) select the statistic valueassociated with the defined current value from the respective level ofthe controller per-level statistics; (m) output the selected statisticvalue to a transformed input dataset to replace the defined currentvalue; (n) repeat (k) to (m) with each remaining variable of the one ormore flow variables as the current variable; and (o) repeat (i) to (n)with each remaining observation vector from the subset of the inputdataset distributed to the respective worker computing device.
 43. Thecomputing system of claim 42, wherein a plurality of transformation flowdefinitions is received, wherein the worker computer-readableinstructions further cause the respective worker computing device to:(p) select a transformation flow definition from the plurality oftransformation flow definitions as a current transformation flow after(a) and before (b), wherein the variable is selected from the one ormore flow variables of the current transformation flow; and (q) repeat(b) to (g) with each remaining transformation flow definition of theplurality of transformation flow definitions as the currenttransformation flow after (g) and before (h), wherein the currentvariable selected in (b) is selected from the one or more flow variablesof the current transformation flow and each remaining variable in (g) isselected from the one or more flow variables of the currenttransformation flow, wherein (p) and (q) are further repeated in (h).44. The computing system of claim 43, wherein a first variable of afirst transformation flow definition of the plurality of transformationflow definitions and a second variable of a second transformation flowdefinition of the plurality of transformation flow definitions areidentical variables.
 45. The computing system of claim 43, wherein theworker computer-readable instructions further cause the respectiveworker computing device to: (r) select the transformation flowdefinition from the plurality of transformation flow definitions as thecurrent transformation flow after (i) and before (j), wherein thevariable is selected from the one or more flow variables of the currenttransformation flow; and (s) repeat (j) to (n) with each remainingtransformation flow definition of the plurality of transformation flowdefinitions as the current transformation flow after (n) and before (o),wherein the current variable selected in (j) is selected from the one ormore flow variables of the current transformation flow and eachremaining variable in (n) is selected from the one or more flowvariables of the current transformation flow, wherein (r) and (s) arefurther repeated in (o).
 46. The computing system of claim 42, whereindefining controller per-level statistics from the defined per-levelstatistics comprises: selecting a first worker computing device from theplurality of worker computing devices as a current worker computingdevice; for each level of the per-level statistics received from thecurrent worker computing device, copying a respective level and thestatistic value associated with the respective level for each variableof the one or more flow variables to the controller per-levelstatistics; selecting a next worker computing device from the pluralityof worker computing devices as the current worker computing device; (aa)for each level of the per-level statistics and for each variable of theone or more flow variables received from the current worker computingdevice, determining if the respective level of each variable of the oneor more flow variables is a new level of the controller per-levelstatistics or an existing level of the controller per-level statistics;when the respective level is the new level of the defined controllerper-level statistics, copying the respective level and the statisticvalue associated with the respective level to the controller per-levelstatistics for a respective variable of the one or more flow variables;and when the respective level is the existing level of the controllerper-level statistics, updating the statistic value associated with theexisting level of the controller per-level statistics for the respectivevariable of the one or more flow variables based on the statistic valueassociated with the respective level received from the current workercomputing device; and repeating (aa) with each remaining workercomputing device of the plurality of worker computing devices as thecurrent worker computing device.
 47. The computing system of claim 42,wherein, after (h) and before requesting transformation of each variableof the one or more flow variables by each of the plurality of workercomputing devices using the controller per-level statistics, thecontroller computer-readable instructions further cause the controllercomputing device to compute a global standardized per-level statisticvalue from the defined controller per-level statistics and to apply ashrinkage factor to the computed global standardized per-level statisticvalue for each level of the defined controller per-level statistics. 48.The computing system of claim 42, wherein, after (h) and beforerequesting transformation of each variable of the one or more flowvariables by each of the plurality of worker computing devices using thedefined controller per-level statistics, the controllercomputer-readable instructions further cause the controller computingdevice to request clustering of each level of the defined controllerper-level statistics of each variable of the one or more flow variablesby the plurality of worker computing devices to define a clusteridentifier for each level.
 49. The computing system of claim 42, whereinthe statistic value is updated in (f) using$M_{p,\zeta} = {M_{p,\zeta_{1}} + {\sum\limits_{k = 1}^{p - 2}\; {\begin{pmatrix}k \\p\end{pmatrix}{M_{{p - k},\zeta_{1}}( \frac{- \delta}{n} )}^{k}}} + {( \frac{( {n - 1} )\delta}{n} )^{p}\lbrack {1 - ( \frac{- 1}{n - 1} )^{p - 1}} \rbrack}}$where p is a predefined number of features to generate for the existinglevel, δ=y−μ₁, where y is a target value of the existing level, and μ₁is a mean value, n is a number of observation values included for theexisting level, M is the statistic value, ζ₁ indicates the statisticvalue without a contribution from the defined current value, and ζindicates the statistic value with the contribution from the definedcurrent value.
 50. A method of transforming variable values in an inputdataset using a plurality of high-cardinality transformation flowdefinitions applied in parallel, the method comprising: receiving atransformation flow definition, wherein the transformation flowdefinition includes one or more flow variables and a transformationmethod to apply to the one or more flow variables, wherein each flowvariable of the one or more flow variables is a high-cardinalityvariable based on a number of unique values of a respective flowvariable of the one or more flow variables; requesting, by a controllerdevice, per-level statistics analysis of an input dataset by each of aplurality of worker computing devices, wherein the input dataset isdistributed across the plurality of worker computing devices; (a)reading, by a worker computing device of the plurality of workercomputing devices, an observation vector from a subset of the inputdataset distributed to a respective worker computing device to define afirst variable value for each variable of the one or more flowvariables; (b) selecting, by the worker computing device, a variable ofthe one or more flow variables as a current variable; (c) defining, bythe worker computing device, a current value equal to the defined firstvariable value for the current variable; (d) determining, by the workercomputing device, if the defined current value is a new level or anexisting level for the current variable; (e) when the defined currentvalue is the new level, initializing, by the worker computing device, astatistic value in association with the new level based on the definedcurrent value and the transformation method to define per-levelstatistics for the current variable; (f) when the defined current valueis the existing level, updating, by the worker computing device, thestatistic value from the defined per-level statistics for the currentvariable in association with the existing level based on the definedcurrent value and the transformation method; (g) repeating, by theworker computing device, (c) to (f) with each remaining variable of theone or more flow variables as the current variable; (h) repeating, bythe worker computing device, (a) to (g) with each remaining observationvector from the subset of the input dataset distributed to the workercomputing device, wherein (a) to (h) is performed by each workercomputing device of the plurality of worker computing devices as theworker computing device; receiving, by the controller device, thedefined per-level statistics for each variable of the one or more flowvariables from each worker computing device of the plurality of workercomputing devices, wherein each level of the defined per-levelstatistics has an associated statistic value; defining, by thecontroller device, controller per-level statistics from the definedper-level statistics for each variable of the one or more flow variablesreceived from each worker computing device of the plurality of workercomputing devices; requesting, by the controller device, transformationof each variable of the one or more flow variables by each of theplurality of worker computing devices using the controller per-levelstatistics; (i) repeating, by the worker computing device, the read ofthe observation vector from the subset of the input dataset distributedto the worker computing device to define the first variable value foreach variable of the one or more flow variables, (j) selecting, by theworker computing device, a second variable of the one or more flowvariables as the current variable; (k) defining, by the worker computingdevice, the current value equal to the defined first variable value forthe current variable; (l) selecting, by the worker computing device, thestatistic value associated with the defined current value from therespective level of the controller per-level statistics; (m) outputting,by the worker computing device, the selected statistic value to atransformed input dataset to replace the defined current value; (n)repeating, by the worker computing device, (k) to (m) with eachremaining variable of the one or more flow variables as the currentvariable; and (o) repeating, by the worker computing device, (i) to (n)with each remaining observation vector from the subset of the inputdataset distributed to the respective worker computing device, wherein(i) to (o) is performed by each worker computing device of the pluralityof worker computing devices as the respective worker computing device.51. The method of claim 50, wherein a plurality of transformation flowdefinitions is received.
 52. The method of claim 51, further comprising:(p) selecting, by the worker computing device, a transformation flowdefinition from the plurality of transformation flow definitions as acurrent transformation flow after (a) and before (b), wherein thevariable is selected from the one or more flow variables of the currenttransformation flow; and (q) repeating, by the worker computing device,(b) to (g) with each remaining transformation flow definition of theplurality of transformation flow definitions as the currenttransformation flow after (g) and before (h), wherein the currentvariable selected in (b) is selected from the one or more flow variablesof the current transformation flow and each remaining variable in (g) isselected from the one or more flow variables of the currenttransformation flow, wherein (p) and (q) are further repeated in (h).53. The method of claim 52, wherein a first variable of a firsttransformation flow definition of the plurality of transformation flowdefinitions and a second variable of a second transformation flowdefinition of the plurality of transformation flow definitions areidentical variables.
 54. The method of claim 53, wherein thetransformation method to apply to the first variable of the firsttransformation flow definition is different from the transformationmethod to apply to the second variable of the second transformation flowdefinition.
 55. The method of claim 52, further comprising: (r)selecting, by the worker computing device, the transformation flowdefinition from the plurality of transformation flow definitions as thecurrent transformation flow after (i) and before (j), wherein thevariable is selected from the one or more flow variables of the currenttransformation flow; and (s) repeating, by the worker computing device,(j) to (n) with each remaining transformation flow definition of theplurality of transformation flow definitions as the currenttransformation flow after (n) and before (o), wherein the currentvariable selected in (j) is selected from the one or more flow variablesof the current transformation flow and each remaining variable in (n) isselected from the one or more flow variables of the currenttransformation flow, wherein (r) and (s) are further repeated in (o).56. The method of claim 50, wherein defining controller per-levelstatistics from the defined per-level statistics comprises: selecting,by the controller device, a first worker computing device from theplurality of worker computing devices as a current worker computingdevice; for each level of the per-level statistics received from thecurrent worker computing device, copying, by the controller device, arespective level and the statistic value associated with the respectivelevel for each variable of the one or more flow variables to definecontroller per-level statistics; selecting, by the controller device, anext worker computing device from the plurality of worker computingdevices as the current worker computing device; (aa) for each level ofthe per-level statistics and for each variable of the one or more flowvariables received from the current worker computing device,determining, by the controller device, if the respective level of eachvariable of the one or more flow variables is a new level of the definedcontroller per-level statistics or an existing level of the controllerper-level statistics; when the respective level is the new level of thedefined controller per-level statistics, copying, by the controllerdevice, the respective level and the statistic value associated with therespective level to the defined controller per-level statistics for arespective variable of the one or more flow variables; and when therespective level is the existing level of the controller per-levelstatistics, updating, by the controller device, the statistic valueassociated with the existing level of the controller per-levelstatistics for the respective variable of the one or more flow variablesbased on the statistic value associated with the respective levelreceived from the current worker computing device; repeating (aa), bythe controller device, with each remaining worker computing device ofthe plurality of worker computing devices as the current workercomputing device.
 57. The method of claim 50, further comprisingcomputing, by the worker computing device, a hash value using hash basedlevel compression applied to the first variable value for each variableof the one or more flow variables based on the transformation flowdefinition to replace the first variable value with the computed hashvalue.
 58. The method of claim 50, further comprising, after (h) andbefore requesting transformation of each variable of the one or moreflow variables by each of the plurality of worker computing devicesusing the defined controller per-level statistics, computing, by thecontroller device, a global standardized per-level statistic value fromthe controller per-level statistics and applying, by the controllerdevice, a shrinkage factor to the computed global standardized per-levelstatistic value for each level of the defined controller per-levelstatistics.
 59. The method of claim 50, further comprising, after (h)and before requesting transformation of each variable of the one or moreflow variables by each of the plurality of worker computing devicesusing the defined controller per-level statistics, requesting, by thecontroller device, clustering-of each level of the defined controllerper-level statistics of each variable of the one or more flow variablesby the plurality of worker computing devices to define a clusteridentifier for each level.
 60. The method of claim 50, wherein thestatistic value is updated in (0 using$M_{p,\zeta} = {M_{p,\zeta_{1}} + {\sum\limits_{k = 1}^{p - 2}\; {\begin{pmatrix}k \\p\end{pmatrix}{M_{{p - k},\zeta_{1}}( \frac{- \delta}{n} )}^{k}}} + {( \frac{( {n - 1} )\delta}{n} )^{p}\lbrack {1 - ( \frac{- 1}{n - 1} )^{p - 1}} \rbrack}}$where p is a predefined number of features to generate for the existinglevel, δ=y−μ₁, where y is a target value of the existing level, and μ₁is a mean value, n is a number of observation values included for theexisting level, M is the statistic value, ζ₁ indicates the statisticvalue without a contribution from the defined current value, and ζindicates the statistic value with the contribution from the definedcurrent value.